| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-01 | |||
| 15:58:46 | dansmith | even still, unless it's writing to the disk when the destroy happens, it wouldn't likely do any damage (and shouldn't really anyway) | |
| 15:59:12 | dansmith | don't we get the console of the guest we fail to ssh into if we do? unless it's sitting at a panic or some unhappy state, I'd not be concerned | |
| 15:59:28 | mriedem | yes we do | |
| 16:00:16 | sean-k-mooney | mriedem: are you suspecting disk curruption? | |
| 16:00:19 | mriedem | there are several guest ssh failure bugs in the gate right now, many different reasons (dhcp lease fails, out of space on the guest disk [growroot fails], kernel panic) | |
| 16:00:28 | sean-k-mooney | due to an agressive shutdown? | |
| 16:00:29 | mriedem | i'm grasping at straws | |
| 16:01:08 | mriedem | in one case that i caught in the gate where the 60 second shutdown timed out, i got the guest console log http://paste.openstack.org/show/752002/ | |
| 16:01:25 | mriedem | cdent wondered (in the bug) if the metadata api retry loop in the guest was making it ignore the shutdown request | |
| 16:01:32 | mriedem | https://bugs.launchpad.net/nova/+bug/1829896 | |
| 16:01:32 | openstack | Launchpad bug 1829896 in OpenStack Compute (nova) "libvirt: "Instance failed to shutdown in 60 seconds." in the gate" [Undecided,New] | |
| 16:02:05 | sean-k-mooney | well in that case it got a dhcp lease but then could not hit the metadata service | |
| 16:02:10 | mriedem | i'm also wondering if there are ideas on figuring out what is eating up all of the disk in these guests when growroot fails | |
| 16:02:17 | sean-k-mooney | so that looks like a neutron issue with the metadata proxy | |
| 16:03:05 | sean-k-mooney | if glean/cloud init is waithing for the metadata then | |
| 16:03:11 | sean-k-mooney | the ssh service may not have started | |
| 16:03:45 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: Add 'path' query parameter to console access url https://review.opendev.org/692573 | |
| 16:06:30 | lyarwood | mriedem: yup, working on the other change now thanks | |
| 16:07:42 | sean-k-mooney | mriedem: i do know we have had issue with hot plug if the early init in the guest takes too long | |
| 16:08:05 | mriedem | lyarwood: i'm sort of inclined to say you guys (RH) should keep that one downstream | |
| 16:08:06 | sean-k-mooney | but we also see Starting acpid: OK | |
| 16:08:17 | mriedem | since i don't see people clamoring for using the bleeding edge novnc stuff back on queens | |
| 16:09:06 | mriedem | i'm not like -2, but those are my "feelings" | |
| 16:09:15 | mriedem | what you humans would call feelings anyway | |
| 16:09:53 | lyarwood | haha yeah no issues, thought I'd at least post it upstream first to see if anyone wanted it | |
| 16:10:02 | sean-k-mooney | ah ha proof mriedem is actully an irc bot !!! | |
| 16:10:54 | sean-k-mooney | mriedem: on other random straw is there seams to be very littly entropy in the guest | |
| 16:10:59 | sean-k-mooney | random: dd urandom read with 31 bits of entropy available | |
| 16:12:14 | sean-k-mooney | the topic of enabling the virtio random number generator by default has come up before. i wonder if it could be related to that. | |
| 16:14:09 | mriedem | i seem to remember a recentish patch of kashyap's related to this | |
| 16:15:21 | mriedem | i'm just thinking of this https://review.opendev.org/#/c/577385/ | |
| 16:16:17 | sean-k-mooney | ya there is that one but we also talked a little about adding the random number gereate to the libvirt xml by defualt rather then needing to set the image property | |
| 16:16:18 | mriedem | we don't use that in our devstack guests though since we don't have hw_rng_model=virtio in the image | |
| 16:16:24 | mriedem | oh | |
| 16:16:37 | mriedem | well, i have to go to what you humans would call lunch now | |
| 16:28:35 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: Reduce scope of 'path' query parameter to noVNC consoles https://review.opendev.org/692581 | |
| 17:19:32 | dansmith | cripes | |
| 17:19:42 | dansmith | cache notifications patch is gonna fail again | |
| 17:21:15 | openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.opendev.org/602201 | |
| 17:34:41 | mriedem | dansmith: it's because of your sin | |
| 17:35:59 | efried | dansmith: I'm +A on mriedem's CD doc patch, but the predecessor (link shuffle) needs to be sent as well | |
| 17:36:07 | efried | https://review.opendev.org/#/c/692569/1 | |
| 17:36:34 | dansmith | efried: oh sorry I missed that earlier I guess | |
| 17:36:46 | dansmith | I dun getified it | |
| 17:40:13 | mriedem | dansmith: i looked at a dstat from one of the 'timed out waiting for cell' failed jobs and the only thing around that time is a spike in paging and networking which doesn't tell me much | |
| 17:40:32 | dansmith | mriedem: yeah I looked pretty in depth at one dstat from a cell fail job | |
| 17:40:36 | dansmith | and it looked overly normal to me | |
| 17:40:53 | dansmith | mysql wasn't spiking that I could tell, it was the big memory user, but only like 400MB at the time | |
| 17:41:02 | dansmith | IO wasn't crazy, etc | |
| 17:41:30 | mriedem | yeah nothing obvious like the 0 i/ops in the other thing earlier today | |
| 17:42:06 | dansmith | I didn't catch the zero iops thing | |
| 17:42:08 | mriedem | and there is no rabbit involved b/c we're going straight to the db yeah? | |
| 17:42:14 | dansmith | you were assuming zero iops meant bad? | |
| 17:42:30 | mriedem | the 0 iops thing was related to another bug in the ML where we get long timeouts | |
| 17:42:42 | mriedem | which triggered the resize long_rpc_timeout discussion | |
| 17:43:08 | mriedem | and a very obvious message in the mysql error log | |
| 17:43:29 | dansmith | you mean zero iops for a long period of time or something I assume/ | |
| 17:43:35 | mriedem | 3 minutes yeah | |
| 17:43:54 | mriedem | while placement was processing a POST /allocations request | |
| 17:43:57 | dansmith | okay, yeah, one or two samples with zero iops wouldn't be alarming to me, but minutes of it, for sure | |
| 17:46:49 | mriedem | and https://bugs.launchpad.net/nova/+bug/1825584 shouldn't be related to the scheduler timing out waiting for a response from cell1 b/c there is no uwsgi involved there | |
| 17:46:49 | openstack | Launchpad bug 1825584 in OpenStack Compute (nova) "eventlet monkey-patching breaks AMQP heartbeat on uWSGI" [Undecided,Confirmed] | |
| 17:46:53 | mriedem | just scheduler running with eventlet | |
| 18:30:42 | openstackgerrit | Merged openstack/nova stable/rocky: libvirt: Ignore volume exceptions during post_live_migration https://review.opendev.org/691283 | |
| 18:36:07 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: libvirt: Ignore volume exceptions during post_live_migration https://review.opendev.org/691284 | |
| 18:48:25 | mriedem | interesting that we don't have a task_state change while confirming a resized server | |
| 18:48:35 | mriedem | to prevent other actions from happening while still confirming and cleaning up the source host | |
| 18:57:59 | mriedem | but we do have task_states.RESIZE_REVERTING for reverting a resize | |
| 19:36:52 | openstackgerrit | Merged openstack/nova master: doc: link to nova code review guide from dev policies https://review.opendev.org/692569 | |
| 19:36:57 | openstackgerrit | Merged openstack/nova master: Document CD mentality policy for nova contributors https://review.opendev.org/692572 | |
| 19:46:58 | mriedem | dansmith: MessagingTimeouts in nova-next in the latest failure in https://review.opendev.org/#/c/691390/ | |
| 19:47:07 | mriedem | looks like just a slow node? | |
| 19:47:24 | mriedem | dstat shows ram usage around 6.5gb for most of the run | |
| 19:47:40 | mriedem | well, at least around the time it was timing out | |
| 19:48:44 | mriedem | load average really spikes around the time of the timeouts | |
| 19:59:37 | sean-k-mooney | mriedem: you ruled out memory right | |
| 20:00:15 | sean-k-mooney | i allocated a little too many hugpage when playing with dpdk today and when i almost exausted memory in the vm i was getting big cpu spikes | |
| 20:00:46 | sean-k-mooney | like all core 100% for 3 secons then normal then spike then normal | |
| 20:08:15 | mriedem | peakmemtracker shows mysql as the top memory user but it doesn't look like it's running out | |
| 20:08:27 | mriedem | should be 8gb of ram on the node | |
| 20:08:40 | sean-k-mooney | ya you are proably fine | |
| 20:09:32 | sean-k-mooney | i had no swap and allcoated 4G of hugepages instead of 2 by mistake so i was running out | |
| 20:21:14 | efried | jroll: You available to brainstorm some of this vTPM stuff? | |
| 20:21:50 | jroll | efried: maybe? | |
| 20:22:05 | jroll | I'm not really an expert on the TPM bits, but I have a few minutes to chat, yes | |
| 20:22:40 | efried | jroll: I'm just trying to understand how the nova flow needs to work. | |
| 20:22:55 | jroll | sure, I can try to help. | |
| 20:23:18 | efried | If each boot needs to be able to specify an independent value, that's a bigger change than other things. | |
| 20:24:23 | efried | Basically what I'm trying to figure out is where the value of `secret` comes from in | |
| 20:24:23 | efried | <tpm model='tpm-tis'> | |
| 20:24:23 | efried | <backend type='emulator' version='2.0'> | |
| 20:24:23 | efried | <encryption secret='6dd3e4a5-1d76-44ce-961f-f119f5aad935'/> | |
| 20:24:23 | efried | </backend> | |
| 20:24:23 | efried | </tpm> | |
| 20:24:35 | jroll | right | |
| 20:24:43 | jroll | so take the fact that's it's a tpm out of your head | |
| 20:24:51 | efried | hah, that's easy enough | |
| 20:24:56 | efried | I can't even spell tpm yet. | |
| 20:24:58 | jroll | basically, there's a file for the instance on the hypervisor's disk, which needs to be encrypted | |
| 20:25:05 | jroll | to encrypt a file, you need a secret | |