| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-01-29 | |||
| 13:09:07 | kashyap | jaosorior: Hang on, though. | |
| 13:09:13 | jaosorior | kashyap: what about connections from nova? | |
| 13:09:16 | kashyap | First, is that a production instance? | |
| 13:09:26 | jaosorior | yes | |
| 13:09:35 | jaosorior | well... not where I'm testing all this stuff :D | |
| 13:09:37 | kashyap | jaosorior: But know that: restarting `libvirtd` will _not_ bring your instances down. | |
| 13:09:40 | jaosorior | but that's the goal, yes | |
| 13:11:24 | kashyap | jaosorior: BTW, the SIGHUP config is ill-defined in the `libvirtd` man page :-( | |
| 13:11:39 | kashyap | It says: "On receipt of SIGHUP libvirtd will reload its configuration." | |
| 13:11:46 | jaosorior | right | |
| 13:11:50 | kashyap | But doesn't expand what is "its configuration". | |
| 13:12:02 | kashyap | I'll get that sorted w/ the libvirt upstream to expand and elaborate. | |
| 13:12:03 | jaosorior | TLS certs/keys are configuration IMO :D | |
| 13:12:59 | kashyap | jaosorior: Okay, so you're using TLS. | |
| 13:13:42 | kashyap | And you want to pick up renewed certificates. Then problem is not libvirtd, I'm afraid — QEMU is the real blocker | |
| 13:14:27 | jaosorior | what are my options? | |
| 13:14:31 | kashyap | Because it (QEMU) can't reload certificates on the fly. (And also QEMU can't just be "restarted") | |
| 13:14:51 | kashyap | Let me check with libvirt folks. | |
| 13:15:37 | jaosorior | kashyap: thanks! | |
| 13:16:20 | kashyap | jaosorior: So, your only viable option is a bit heavy: live migration of the QEMU instance(s). | |
| 13:16:21 | kashyap | (I.e. Nova instances) | |
| 13:16:48 | ioni | where the certs are used? | |
| 13:19:21 | kashyap | ioni: For TLS for various Nova services, I presume | |
| 13:19:47 | kashyap | jaosorior: Related, recently we merged this: http://git.openstack.org/cgit/openstack/nova/tree/doc/source/admin/secure-live-migration-with-qemu-native-tls.rst | |
| 13:20:04 | kashyap | There is a nicer rendered version | |
| 13:20:53 | kashyap | jaosorior: http://logs.openstack.org/83/631283/3/check/openstack-tox-docs/615979c/html/admin/secure-live-migration-with-qemu-native-tls.html | |
| 13:22:08 | jaosorior | kashyap: I'm a little confused though. the TLS implementation for connections to libvirtd is not the same one then as the one for qemu, right? | |
| 13:22:27 | kashyap | (But the above URL doesn't solve the existing problem, though. It is "future goodness", when you upgrade.) | |
| 13:23:20 | kashyap | jaosorior: Are you using the 'live_migration_tunnelled' Nova config? | |
| 13:24:05 | kashyap | jaosorior: They're somewhat connected: when you use TLS with libvirt, it is passing the TLS creds under the hood to QEMU. | |
| 13:24:18 | jaosorior | kashyap: we are, yeah | |
| 13:24:19 | kashyap | jaosorior: I can tell more details, if you post your QEMU command-line of your Nova instance (from the TLS env) | |
| 13:24:30 | jaosorior | kashyap: sure | |
| 13:24:55 | kashyap | I need to grab a quick bite (not going to eat in front of the computer, as a rule). Be back in 15 mins | |
| 13:25:13 | jaosorior | kashyap: sure. Good time for me to make some coffee | |
| 13:32:24 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Skip checking of target_dev for vhostuser https://review.openstack.org/633734 | |
| 13:43:24 | kashyap | jaosorior: Let me know when you have the output. | |
| 13:50:20 | kashyap | jaosorior: But regardless, this requires support in QEMU and libvirt. I'll file RFEs w/ relevant upstreams | |
| 13:50:35 | jaosorior | kashyap: that would be great! (sorry, I'm in a meeting) | |
| 15:06:51 | gibi | jaypipes, mriedem: thanks for the reviews on the bandwidth series. I will propose the fup patch based on your comments | |
| 15:07:31 | jaypipes | gibi: ack. will try to get through remaining patches later. | |
| 15:08:13 | gibi | jaypipes: I saw your -1 in the middle. I will need to get back to dansmith about the second microversion | |
| 15:09:54 | mriedem | gibi: yw, glad to help move this along | |
| 15:10:20 | mriedem | also hopefully we can maybe merge some code yet this month https://review.openstack.org/#/c/633566/ | |
| 15:15:09 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: API microversion 2.68: Handles Down Cells https://review.openstack.org/591657 | |
| 15:17:44 | gibi | mriedem: yeah, the gate is not on my side :) | |
| 15:22:29 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add configuration of maximum disk devices to attach https://review.openstack.org/616777 | |
| 15:23:19 | mriedem | i'm +2 on ^ now | |
| 15:23:23 | mriedem | and the one below | |
| 15:24:33 | jaypipes | mriedem: amen on the merging code comment. | |
| 15:26:21 | kashyap | mriedem: I'm only 3 months late on your question here: https://review.openstack.org/#/c/506720/ | |
| 15:26:39 | kashyap | (In short, I'm planning to revive that Secure Boot spec for Train.) | |
| 15:29:26 | kashyap | I posted a few interim updates there based on the completed work in OVMF, QEMU, libvirt & related tooling. I didn't post one since last August due to the libvirt bug still being in ASSIGNED state: https://bugzilla.redhat.com/show_bug.cgi?id=1605127 | |
| 15:29:27 | openstack | bugzilla.redhat.com bug 1605127 in libvirt "RFE: QEMU firmware metadata format - libvirt support" [Unspecified,Assigned] - Assigned to phrdina | |
| 15:33:30 | mriedem | jangutter: i looked at the evacuate network-vif-plugged timeout errors in https://review.openstack.org/#/c/571325/ - tldr is it looks like we get the event before we're ready for it, which is weird | |
| 15:34:20 | jangutter | mriedem: think it's something transient or still surviving in master branches? | |
| 15:34:39 | mriedem | i assume it's transient otherwise that job would be 100% failure | |
| 15:34:59 | mriedem | would be useful if we logged when we register the event callbacks | |
| 15:37:04 | mriedem | oh we do | |
| 15:37:05 | mriedem | Jan 29 01:11:55.567189 ubuntu-xenial-rax-ord-0002201755 nova-compute[15903]: DEBUG nova.compute.manager [None req-252304a1-6eff-4ff3-aa4d-b4e0ab87601c demo admin] [instance: 2e82ddcd-75b8-4a41-8ecd-ce175adbdc67] Preparing to wait for external event network-vif-plugged-e241f79f-fb0d-4961-b0c8-aea9de2755bf {{(pid=15903) prepare_for_instance_event /opt/stack/new/nova/nova/compute/manager.py:325}} | |
| 15:38:35 | mriedem | which is too late | |
| 15:39:41 | mriedem | Jan 29 01:11:49.707004 ubuntu-xenial-rax-ord-0002201755 devstack@n-api.service[22319]: DEBUG nova.api.openstack.wsgi [req-3ed0ada1-7328-4d5a-a3ea-da34dcdb252d req-1f5aeede-83c6-44d5-afd4-b5435e71d61f service nova] Action: 'create', calling method: <bound method ServerExternalEventsController.create of <nova.api.openstack.compute.server_external_events.ServerExternalEventsController object at 0x7fb0133aa050>>, body: {"e | |
| 15:39:41 | mriedem | s": [{"status": "completed", "tag": "e241f79f-fb0d-4961-b0c8-aea9de2755bf", "name": "network-vif-plugged", "server_uuid": "2e82ddcd-75b8-4a41-8ecd-ce175adbdc67"}]} {{(pid=22322) _process_stack /opt/stack/new/nova/nova/api/openstack/wsgi.py:520}} | |
| 15:41:25 | jangutter | mriedem: nicely reversed causality there. | |
| 15:43:29 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Timeout%20waiting%20for%20%5B('network-vif-plugged'%5C%22%20AND%20message%3A%5C%22for%20instance%20with%20vm_state%20error%20and%20task_state%20rebuild_spawning.%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d | |
| 15:46:36 | mnaser | is there any stable cores around to +w this? https://review.openstack.org/#/c/619352/ | |
| 15:47:07 | mriedem | jangutter: https://bugs.launchpad.net/nova/+bug/1813789 | |
| 15:47:08 | openstack | Launchpad bug 1813789 in OpenStack Compute (nova) "Evacuate test intermittently fails with network-vif-plugged timeout exception" [Medium,Confirmed] | |
| 16:08:10 | mriedem | jaypipes: since melwitt is out this week i replied on https://review.openstack.org/#/c/632904/ for that TooManyDiskDevices API error check | |
| 16:08:28 | mriedem | tl;dr there was already a lot of discussion between myself, melwitt, cdent and edleafe on that 403 | |
| 16:09:16 | mriedem | i personally don't care if we change to 409, but i don't really think 400 is correct | |
| 16:11:57 | cdent | yeah, 409 is most correct, but 403 is consistent | |
| 16:17:40 | efried | mriedem: Where's the code that converts ComputeDriver.capabilities into traits on the compute node RP?? | |
| 16:18:20 | mriedem | efried: https://review.openstack.org/#/c/538498/ | |
| 16:18:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Fup for the bandwidth resource provider series https://review.openstack.org/633776 | |
| 16:18:32 | efried | o | |
| 16:18:50 | efried | Why hasn't this merged yet <== because some a-hole -1'd it. | |
| 16:19:00 | mriedem | because i haven't updated it | |
| 16:19:05 | gibi | jaypipes, mriedem: here is the followup fixing comments in patches that are approved https://review.openstack.org/633776 | |
| 16:19:26 | mriedem | gibi: ack | |
| 16:19:46 | efried | mriedem: Cool beans, just knew I had seen said code somewhere, but couldn't find it in the master branch. Thanks. | |
| 16:21:35 | kashyap | jangutter: Where do you see the 'run-devstack' taking 30 minutes here: http://logs.openstack.org/80/630980/7/gate/tempest-full/5b6ebba/ara-report/ | |
| 16:22:12 | kashyap | jangutter: The one I found actually successfully "FINISHED" | |
| 16:22:28 | kashyap | (Although taking 30 mins & completing are not mutually exclusive.) | |
| 16:22:57 | jangutter | kashyap: in the devstack-tempest.yaml line, click on the "16 Tasks" button to pull them down. The run-devstack : Run devstack task went over 30 minutes. | |
| 16:23:20 | kashyap | jangutter: Right, there the status of 'run-devstack' task is "CHANGED". | |
| 16:23:29 | jangutter | kashyap: if you click on the parameters for the plays, you'll also see "timeout 30" | |
| 16:24:27 | kashyap | jangutter: Right, there I see it's 30 mins. And when I click on its 'Status' ("CHANGED" - what does it even mean), the job finished. | |
| 16:24:39 | kashyap | So I don't see any action item there. | |
| 16:24:42 | jangutter | kashyap: the playbook was interrupted, so the ARA results are inconsistent | |
| 16:24:47 | kashyap | Hmm | |
| 16:25:08 | kashyap | Just want to put this change out of its misery; it's been in time-out/recheck hell for 9 days | |
| 16:26:54 | jangutter | kashyap: http://logs.openstack.org/80/630980/7/gate/tempest-full/5b6ebba/job-output.txt.gz#_2019-01-29_00_14_28_138371 <--- this Ansible telling you the play timed out. | |
| 16:27:19 | kashyap | Ah, _there_ is the specific piece of hay in the haystack! | |
| 16:27:30 | jangutter | kashyap: is the recheck still in the gate? | |
| 16:27:39 | kashyap | Yeah, 8 hours and running. | |
| 16:28:03 | kashyap | Or something like that. But it doesn't bother me much, I've got a more than a few irons in the fire. | |
| 16:28:17 | kashyap | Just that it's annoying to wake up each morning to see that thing time out. | |
| 16:28:32 | jangutter | kashyap: the sarlacc pit is slow, but thorough. (plagiarizing and butchering jaypipes). | |
| 16:28:37 | openstackgerrit | Adrian Chiris proposed openstack/nova master: Add free for claimed, allocated devices https://review.openstack.org/616120 | |