| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-01-29 | |||
| 09:43:56 | jangutter | kashyap: yep, I can almost 100% guarantee that there isn't. | |
| 09:44:10 | jangutter | kashyap: (a fix related to your patch, I mean). | |
| 09:44:32 | kashyap | I don't see any failures here either: http://logs.openstack.org/80/630980/7/gate/tempest-full/5b6ebba/testr_results.html.gz | |
| 09:45:35 | jangutter | kashyap: I think devstack took too long to spin up. | |
| 09:50:14 | jangutter | kashyap: If you look at http://logs.openstack.org/80/630980/7/gate/tempest-full/5b6ebba/ara-report/ <--- it took 30 minutes for the run-devstack task. | |
| 09:50:54 | jangutter | kashyap: some of these failures require a skill level that waay beyond my ability to comprehend. | |
| 09:52:49 | kashyap | Hmm | |
| 09:52:56 | kashyap | Thanks for looking; I'll check w/ the -infra folks | |
| 09:56:40 | jangutter | kashyap: yeah, digging into that ara-report, there's a 30 minute timeout on tasks, and the task went too long. Could be that the VM was running under contention or something so there's a very high chance a recheck will succeed. | |
| 09:56:58 | kashyap | Alright, we've had about 6 re-checks | |
| 09:57:13 | kashyap | Another one is in progress; let's see. | |
| 11:34:15 | tinwood | Morning. Is there anyone who could take a look at https://review.openstack.org/#/c/633279/ for me; I needed to add the LXD bit, but not sure if I should have version bumped. Many thanks! | |
| 11:49:49 | gibi | kashyap: I didn't see such tempest timeout recently but not long ago we had problems with tempest slowness and moved some of the tests to tempest-slow | |
| 11:51:15 | ioni | hello guys | |
| 11:51:34 | gibi | kashyap: if you can dig up what are the slowest test in those runs that timed out then we can move some more test to tempest-slow | |
| 11:51:41 | ioni | what's the right way to disable out of date images ? | |
| 11:51:53 | ioni | --deactivate or --private or other way? | |
| 11:52:36 | ioni | because now i'm using --deactive and if an instance that is using this image wants to resize, nova returnes and error that the project doesn't have permission to that specific image | |
| 11:52:40 | ioni | that is now in error state | |
| 11:52:57 | ioni | i'm doing something wrong or is a bug? | |
| 11:55:11 | ioni | the error log: https://paste.xinu.at/b6ZsNA/ | |
| 12:56:36 | jaosorior | kashyap: Hello! Are you the right person to ask a libvirtd related question? | |
| 13:02:03 | kashyap | gibi: Give me a few; check out this: http://logs.openstack.org/80/630980/7/gate/tempest-full/5b6ebba/ara-report/ | |
| 13:02:23 | kashyap | gibi: Pointed out by jangutter earlier; where the 'run-devstack' task was taking 30 minutes | |
| 13:02:33 | kashyap | jaosorior: Hi, you can you ask, I can try :-) | |
| 13:03:18 | jaosorior | kashyap: I'm trying to reload libvirtd, given that it's certificate got rotated. in the man pages it says SIGHUP should do that... but it seems it doesn't... wanted to know if this is a bug, or if there's some other mechanism I could/should use instead. | |
| 13:06:54 | kashyap | jaosorior: Hmm, interesting problem | |
| 13:07:26 | kashyap | jaosorior: Can you afford to do an explicit stop+start of the daemon? | |
| 13:07:58 | kashyap | If so, does that pick up the new certs? | |
| 13:08:48 | jaosorior | kashyap: would that disrupt instances running on the host? | |
| 13:08:57 | kashyap | jaosorior: Good news: no. | |
| 13:09:07 | kashyap | jaosorior: Hang on, though. | |
| 13:09:13 | jaosorior | kashyap: what about connections from nova? | |
| 13:09:16 | kashyap | First, is that a production instance? | |
| 13:09:26 | jaosorior | yes | |
| 13:09:35 | jaosorior | well... not where I'm testing all this stuff :D | |
| 13:09:37 | kashyap | jaosorior: But know that: restarting `libvirtd` will _not_ bring your instances down. | |
| 13:09:40 | jaosorior | but that's the goal, yes | |
| 13:11:24 | kashyap | jaosorior: BTW, the SIGHUP config is ill-defined in the `libvirtd` man page :-( | |
| 13:11:39 | kashyap | It says: "On receipt of SIGHUP libvirtd will reload its configuration." | |
| 13:11:46 | jaosorior | right | |
| 13:11:50 | kashyap | But doesn't expand what is "its configuration". | |
| 13:12:02 | kashyap | I'll get that sorted w/ the libvirt upstream to expand and elaborate. | |
| 13:12:03 | jaosorior | TLS certs/keys are configuration IMO :D | |
| 13:12:59 | kashyap | jaosorior: Okay, so you're using TLS. | |
| 13:13:42 | kashyap | And you want to pick up renewed certificates. Then problem is not libvirtd, I'm afraid — QEMU is the real blocker | |
| 13:14:27 | jaosorior | what are my options? | |
| 13:14:31 | kashyap | Because it (QEMU) can't reload certificates on the fly. (And also QEMU can't just be "restarted") | |
| 13:14:51 | kashyap | Let me check with libvirt folks. | |
| 13:15:37 | jaosorior | kashyap: thanks! | |
| 13:16:20 | kashyap | jaosorior: So, your only viable option is a bit heavy: live migration of the QEMU instance(s). | |
| 13:16:21 | kashyap | (I.e. Nova instances) | |
| 13:16:48 | ioni | where the certs are used? | |
| 13:19:21 | kashyap | ioni: For TLS for various Nova services, I presume | |
| 13:19:47 | kashyap | jaosorior: Related, recently we merged this: http://git.openstack.org/cgit/openstack/nova/tree/doc/source/admin/secure-live-migration-with-qemu-native-tls.rst | |
| 13:20:04 | kashyap | There is a nicer rendered version | |
| 13:20:53 | kashyap | jaosorior: http://logs.openstack.org/83/631283/3/check/openstack-tox-docs/615979c/html/admin/secure-live-migration-with-qemu-native-tls.html | |
| 13:22:08 | jaosorior | kashyap: I'm a little confused though. the TLS implementation for connections to libvirtd is not the same one then as the one for qemu, right? | |
| 13:22:27 | kashyap | (But the above URL doesn't solve the existing problem, though. It is "future goodness", when you upgrade.) | |
| 13:23:20 | kashyap | jaosorior: Are you using the 'live_migration_tunnelled' Nova config? | |
| 13:24:05 | kashyap | jaosorior: They're somewhat connected: when you use TLS with libvirt, it is passing the TLS creds under the hood to QEMU. | |
| 13:24:18 | jaosorior | kashyap: we are, yeah | |
| 13:24:19 | kashyap | jaosorior: I can tell more details, if you post your QEMU command-line of your Nova instance (from the TLS env) | |
| 13:24:30 | jaosorior | kashyap: sure | |
| 13:24:55 | kashyap | I need to grab a quick bite (not going to eat in front of the computer, as a rule). Be back in 15 mins | |
| 13:25:13 | jaosorior | kashyap: sure. Good time for me to make some coffee | |
| 13:32:24 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Skip checking of target_dev for vhostuser https://review.openstack.org/633734 | |
| 13:43:24 | kashyap | jaosorior: Let me know when you have the output. | |
| 13:50:20 | kashyap | jaosorior: But regardless, this requires support in QEMU and libvirt. I'll file RFEs w/ relevant upstreams | |
| 13:50:35 | jaosorior | kashyap: that would be great! (sorry, I'm in a meeting) | |
| 15:06:51 | gibi | jaypipes, mriedem: thanks for the reviews on the bandwidth series. I will propose the fup patch based on your comments | |
| 15:07:31 | jaypipes | gibi: ack. will try to get through remaining patches later. | |
| 15:08:13 | gibi | jaypipes: I saw your -1 in the middle. I will need to get back to dansmith about the second microversion | |
| 15:09:54 | mriedem | gibi: yw, glad to help move this along | |
| 15:10:20 | mriedem | also hopefully we can maybe merge some code yet this month https://review.openstack.org/#/c/633566/ | |
| 15:15:09 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: API microversion 2.68: Handles Down Cells https://review.openstack.org/591657 | |
| 15:17:44 | gibi | mriedem: yeah, the gate is not on my side :) | |
| 15:22:29 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add configuration of maximum disk devices to attach https://review.openstack.org/616777 | |
| 15:23:19 | mriedem | i'm +2 on ^ now | |
| 15:23:23 | mriedem | and the one below | |
| 15:24:33 | jaypipes | mriedem: amen on the merging code comment. | |
| 15:26:21 | kashyap | mriedem: I'm only 3 months late on your question here: https://review.openstack.org/#/c/506720/ | |
| 15:26:39 | kashyap | (In short, I'm planning to revive that Secure Boot spec for Train.) | |
| 15:29:26 | kashyap | I posted a few interim updates there based on the completed work in OVMF, QEMU, libvirt & related tooling. I didn't post one since last August due to the libvirt bug still being in ASSIGNED state: https://bugzilla.redhat.com/show_bug.cgi?id=1605127 | |
| 15:29:27 | openstack | bugzilla.redhat.com bug 1605127 in libvirt "RFE: QEMU firmware metadata format - libvirt support" [Unspecified,Assigned] - Assigned to phrdina | |
| 15:33:30 | mriedem | jangutter: i looked at the evacuate network-vif-plugged timeout errors in https://review.openstack.org/#/c/571325/ - tldr is it looks like we get the event before we're ready for it, which is weird | |
| 15:34:20 | jangutter | mriedem: think it's something transient or still surviving in master branches? | |
| 15:34:39 | mriedem | i assume it's transient otherwise that job would be 100% failure | |
| 15:34:59 | mriedem | would be useful if we logged when we register the event callbacks | |
| 15:37:04 | mriedem | oh we do | |
| 15:37:05 | mriedem | Jan 29 01:11:55.567189 ubuntu-xenial-rax-ord-0002201755 nova-compute[15903]: DEBUG nova.compute.manager [None req-252304a1-6eff-4ff3-aa4d-b4e0ab87601c demo admin] [instance: 2e82ddcd-75b8-4a41-8ecd-ce175adbdc67] Preparing to wait for external event network-vif-plugged-e241f79f-fb0d-4961-b0c8-aea9de2755bf {{(pid=15903) prepare_for_instance_event /opt/stack/new/nova/nova/compute/manager.py:325}} | |
| 15:38:35 | mriedem | which is too late | |
| 15:39:41 | mriedem | Jan 29 01:11:49.707004 ubuntu-xenial-rax-ord-0002201755 devstack@n-api.service[22319]: DEBUG nova.api.openstack.wsgi [req-3ed0ada1-7328-4d5a-a3ea-da34dcdb252d req-1f5aeede-83c6-44d5-afd4-b5435e71d61f service nova] Action: 'create', calling method: <bound method ServerExternalEventsController.create of <nova.api.openstack.compute.server_external_events.ServerExternalEventsController object at 0x7fb0133aa050>>, body: {"e | |
| 15:39:41 | mriedem | s": [{"status": "completed", "tag": "e241f79f-fb0d-4961-b0c8-aea9de2755bf", "name": "network-vif-plugged", "server_uuid": "2e82ddcd-75b8-4a41-8ecd-ce175adbdc67"}]} {{(pid=22322) _process_stack /opt/stack/new/nova/nova/api/openstack/wsgi.py:520}} | |
| 15:41:25 | jangutter | mriedem: nicely reversed causality there. | |
| 15:43:29 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Timeout%20waiting%20for%20%5B('network-vif-plugged'%5C%22%20AND%20message%3A%5C%22for%20instance%20with%20vm_state%20error%20and%20task_state%20rebuild_spawning.%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d | |
| 15:46:36 | mnaser | is there any stable cores around to +w this? https://review.openstack.org/#/c/619352/ | |
| 15:47:07 | mriedem | jangutter: https://bugs.launchpad.net/nova/+bug/1813789 | |
| 15:47:08 | openstack | Launchpad bug 1813789 in OpenStack Compute (nova) "Evacuate test intermittently fails with network-vif-plugged timeout exception" [Medium,Confirmed] | |
| 16:08:10 | mriedem | jaypipes: since melwitt is out this week i replied on https://review.openstack.org/#/c/632904/ for that TooManyDiskDevices API error check | |