Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-01-29
15:06:51 gibi jaypipes, mriedem: thanks for the reviews on the bandwidth series. I will propose the fup patch based on your comments
15:07:31 jaypipes gibi: ack. will try to get through remaining patches later.
15:08:13 gibi jaypipes: I saw your -1 in the middle. I will need to get back to dansmith about the second microversion
15:09:54 mriedem gibi: yw, glad to help move this along
15:10:20 mriedem also hopefully we can maybe merge some code yet this month https://review.openstack.org/#/c/633566/
15:15:09 openstackgerrit Surya Seetharaman proposed openstack/nova master: API microversion 2.68: Handles Down Cells https://review.openstack.org/591657
15:17:44 gibi mriedem: yeah, the gate is not on my side :)
15:22:29 openstackgerrit Matt Riedemann proposed openstack/nova master: Add configuration of maximum disk devices to attach https://review.openstack.org/616777
15:23:19 mriedem i'm +2 on ^ now
15:23:23 mriedem and the one below
15:24:33 jaypipes mriedem: amen on the merging code comment.
15:26:21 kashyap mriedem: I'm only 3 months late on your question here: https://review.openstack.org/#/c/506720/
15:26:39 kashyap (In short, I'm planning to revive that Secure Boot spec for Train.)
15:29:26 kashyap I posted a few interim updates there based on the completed work in OVMF, QEMU, libvirt & related tooling. I didn't post one since last August due to the libvirt bug still being in ASSIGNED state: https://bugzilla.redhat.com/show_bug.cgi?id=1605127
15:29:27 openstack bugzilla.redhat.com bug 1605127 in libvirt "RFE: QEMU firmware metadata format - libvirt support" [Unspecified,Assigned] - Assigned to phrdina
15:33:30 mriedem jangutter: i looked at the evacuate network-vif-plugged timeout errors in https://review.openstack.org/#/c/571325/ - tldr is it looks like we get the event before we're ready for it, which is weird
15:34:20 jangutter mriedem: think it's something transient or still surviving in master branches?
15:34:39 mriedem i assume it's transient otherwise that job would be 100% failure
15:34:59 mriedem would be useful if we logged when we register the event callbacks
15:37:04 mriedem oh we do
15:37:05 mriedem Jan 29 01:11:55.567189 ubuntu-xenial-rax-ord-0002201755 nova-compute[15903]: DEBUG nova.compute.manager [None req-252304a1-6eff-4ff3-aa4d-b4e0ab87601c demo admin] [instance: 2e82ddcd-75b8-4a41-8ecd-ce175adbdc67] Preparing to wait for external event network-vif-plugged-e241f79f-fb0d-4961-b0c8-aea9de2755bf {{(pid=15903) prepare_for_instance_event /opt/stack/new/nova/nova/compute/manager.py:325}}
15:38:35 mriedem which is too late
15:39:41 mriedem Jan 29 01:11:49.707004 ubuntu-xenial-rax-ord-0002201755 devstack@n-api.service[22319]: DEBUG nova.api.openstack.wsgi [req-3ed0ada1-7328-4d5a-a3ea-da34dcdb252d req-1f5aeede-83c6-44d5-afd4-b5435e71d61f service nova] Action: 'create', calling method: <bound method ServerExternalEventsController.create of <nova.api.openstack.compute.server_external_events.ServerExternalEventsController object at 0x7fb0133aa050>>, body: {"e
15:39:41 mriedem s": [{"status": "completed", "tag": "e241f79f-fb0d-4961-b0c8-aea9de2755bf", "name": "network-vif-plugged", "server_uuid": "2e82ddcd-75b8-4a41-8ecd-ce175adbdc67"}]} {{(pid=22322) _process_stack /opt/stack/new/nova/nova/api/openstack/wsgi.py:520}}
15:41:25 jangutter mriedem: nicely reversed causality there.
15:43:29 mriedem http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Timeout%20waiting%20for%20%5B('network-vif-plugged'%5C%22%20AND%20message%3A%5C%22for%20instance%20with%20vm_state%20error%20and%20task_state%20rebuild_spawning.%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d
15:46:36 mnaser is there any stable cores around to +w this? https://review.openstack.org/#/c/619352/
15:47:07 mriedem jangutter: https://bugs.launchpad.net/nova/+bug/1813789
15:47:08 openstack Launchpad bug 1813789 in OpenStack Compute (nova) "Evacuate test intermittently fails with network-vif-plugged timeout exception" [Medium,Confirmed]
16:08:10 mriedem jaypipes: since melwitt is out this week i replied on https://review.openstack.org/#/c/632904/ for that TooManyDiskDevices API error check
16:08:28 mriedem tl;dr there was already a lot of discussion between myself, melwitt, cdent and edleafe on that 403
16:09:16 mriedem i personally don't care if we change to 409, but i don't really think 400 is correct
16:11:57 cdent yeah, 409 is most correct, but 403 is consistent
16:17:40 efried mriedem: Where's the code that converts ComputeDriver.capabilities into traits on the compute node RP??
16:18:20 mriedem efried: https://review.openstack.org/#/c/538498/
16:18:28 openstackgerrit Balazs Gibizer proposed openstack/nova master: Fup for the bandwidth resource provider series https://review.openstack.org/633776
16:18:32 efried o
16:18:50 efried Why hasn't this merged yet <== because some a-hole -1'd it.
16:19:00 mriedem because i haven't updated it
16:19:05 gibi jaypipes, mriedem: here is the followup fixing comments in patches that are approved https://review.openstack.org/633776
16:19:26 mriedem gibi: ack
16:19:46 efried mriedem: Cool beans, just knew I had seen said code somewhere, but couldn't find it in the master branch. Thanks.
16:21:35 kashyap jangutter: Where do you see the 'run-devstack' taking 30 minutes here: http://logs.openstack.org/80/630980/7/gate/tempest-full/5b6ebba/ara-report/
16:22:12 kashyap jangutter: The one I found actually successfully "FINISHED"
16:22:28 kashyap (Although taking 30 mins & completing are not mutually exclusive.)
16:22:57 jangutter kashyap: in the devstack-tempest.yaml line, click on the "16 Tasks" button to pull them down. The run-devstack : Run devstack task went over 30 minutes.
16:23:20 kashyap jangutter: Right, there the status of 'run-devstack' task is "CHANGED".
16:23:29 jangutter kashyap: if you click on the parameters for the plays, you'll also see "timeout 30"
16:24:27 kashyap jangutter: Right, there I see it's 30 mins. And when I click on its 'Status' ("CHANGED" - what does it even mean), the job finished.
16:24:39 kashyap So I don't see any action item there.
16:24:42 jangutter kashyap: the playbook was interrupted, so the ARA results are inconsistent
16:24:47 kashyap Hmm
16:25:08 kashyap Just want to put this change out of its misery; it's been in time-out/recheck hell for 9 days
16:26:54 jangutter kashyap: http://logs.openstack.org/80/630980/7/gate/tempest-full/5b6ebba/job-output.txt.gz#_2019-01-29_00_14_28_138371 <--- this Ansible telling you the play timed out.
16:27:19 kashyap Ah, _there_ is the specific piece of hay in the haystack!
16:27:30 jangutter kashyap: is the recheck still in the gate?
16:27:39 kashyap Yeah, 8 hours and running.
16:28:03 kashyap Or something like that. But it doesn't bother me much, I've got a more than a few irons in the fire.
16:28:17 kashyap Just that it's annoying to wake up each morning to see that thing time out.
16:28:32 jangutter kashyap: the sarlacc pit is slow, but thorough. (plagiarizing and butchering jaypipes).
16:28:37 openstackgerrit Adrian Chiris proposed openstack/nova master: Add free for claimed, allocated devices https://review.openstack.org/616120
16:28:37 openstackgerrit Adrian Chiris proposed openstack/nova master: Allow per-port modification of vnic_type and profile https://review.openstack.org/607365
16:28:38 openstackgerrit Adrian Chiris proposed openstack/nova master: Add get_instance_pci_request_from_vif https://review.openstack.org/619929
16:28:38 openstackgerrit Adrian Chiris proposed openstack/nova master: SR-IOV Live migration indirect port support https://review.openstack.org/620115
16:28:40 mriedem gibi: regarding the microversion for move operations comment in https://review.openstack.org/#/c/630722/ i replied that i don't think we have to make that decision at that point in the series
16:28:58 kashyap jangutter: Hehe
16:29:51 jangutter kashyap: you'll see the tempest tests above that line have got times attached. Not sure if you could also grab them from the testr results, but chances are one of them is a candidate for the -slow set.
16:30:57 jangutter kashyap: I did see your review seems to have hit _different_ snags in the gate each time, so congratulations on being kind of a lightning rod.
16:31:27 kashyap jangutter: Yeah, I see some of the network tests are taking terribly long (test_server_connectivity_reboot)
16:32:02 jangutter kashyap: a heck of a lot of neutron and keystone reviews are getting booted because of what seems to be a sudden tightening up of pep8 enforcement.
16:32:06 kashyap jangutter: Yep, mumble-mumble something about "good deeds".
16:33:37 kashyap jangutter: I see, thanks for the info.
17:23:20 stephenfin Nice docs patch here if anyone fancies taking a look https://review.openstack.org/#/c/626931
17:31:18 kashyap stephenfin: Yes, that's an important point easy to miss.
17:32:49 kashyap +1ed, FWIW.
17:36:15 openstackgerrit Merged openstack/nova master: libvirt: A few miscellaneous items related to "native TLS" https://review.openstack.org/630980
17:36:24 openstackgerrit Merged openstack/nova master: docs: Update references to "QEMU-native TLS" document https://review.openstack.org/631283
17:41:24 jangutter hehehe, kashyap, another timeout http://logs.openstack.org/59/624959/5/gate/tempest-full/ce2253e/job-output.txt.gz#_2019-01-29_17_20_17_952753 but looks like your reviews got merged!
17:42:01 kashyap Very bizarre
17:42:08 kashyap But I won't complain
17:42:27 kashyap That's a good excuse to go cook some dinner.
17:43:17 kashyap jangutter: The difference is time-out was not in the _same_ job ("tempest-full"), which succeeded this time.
18:10:19 openstackgerrit Jack Ding proposed openstack/nova master: Preserve UEFI NVRAM variable store https://review.openstack.org/621646
18:12:36 openstackgerrit Jack Ding proposed openstack/nova master: Improve libvirt image and snapshot handling https://review.openstack.org/616692
18:14:26 openstackgerrit Jack Ding proposed openstack/nova master: Correct instance port binding for rebuilds https://review.openstack.org/603844
18:51:59 tomtom001 hello, I am running openstack queens and I am having trouble getting nova instances to boot using emphemeral storage. I have force_raw_images = false and use_cow_images = true for my settings, would these conflict with that? I also have Host aggregates setting ephemeralcomputestorage = false would this cause a problem for the boot process? When my instances boot they just go to "No bootable
18:52:05 tomtom001 device" nova, cinder, and ceph seem to be all functioning correctly.
18:53:23 tomtom001 While I've found documentation for the nova settings, I have not found any for the Host Aggregate settings and how it affects the storage selection.
19:07:39 mriedem ephemeralcomputestorage isn't a special key in nova,
19:07:55 mriedem presumably you're trying to use that to restrict certain flavors or images to that host aggregate and linking it by the ephemeralcomputestorage key
19:08:20 mriedem e.g. https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#aggregateimagepropertiesisolation and https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#aggregateinstanceextraspecsfilter
19:09:46 tomtom001 mriedem: so if I've done that then I wouldn't be able to use ephemeral on our host aggregates with that key set?
19:10:20 mriedem are you trying to boot from volume?
19:10:47 mriedem i don't know where the "no bootable device" error is coming from - libvirt?
19:11:37 mriedem also not sure what you mean by ephemeral storage - do you mean local storage on the compute node?
19:11:43 mriedem i.e. not volume backed
19:12:24 tomtom001 from the console of the instance, showing no bootable device, ephemeral is not local nova disk, just not volume backed.
19:13:04 mriedem sounds like a problem with the image
19:14:17 tomtom001 The same images work if booted with a volume backing.
19:23:41 openstackgerrit Hang Yang proposed openstack/nova stable/rocky: Fix port dns_name reset https://review.openstack.org/633806

Earlier   Later