| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-15 | |||
| 19:04:00 | efried | blam! | |
| 19:04:03 | efried | thanks bbobrov | |
| 19:04:19 | bbobrov | i will reply to the comments now | |
| 19:04:56 | artom | What happened to aspiers? | |
| 19:21:41 | efried | was wondering same | |
| 19:35:04 | openstackgerrit | Eric Fried proposed openstack/nova master: Use Adapter global_request_id kwarg https://review.opendev.org/670907 | |
| 19:35:15 | cdent | efried: the semaphore thing is indirectly related to https://bugs.launchpad.net/nova/+bug/1835958 (power states not under the same lock, but in the realm of nova-compute performance) | |
| 19:35:15 | openstack | Launchpad bug 1835958 in OpenStack Compute (nova) "Nova sync power state on large clusters causes poor performance" [Undecided,New] | |
| 19:37:04 | efried | cdent: Only slightly related, I've been talking to alex_xu about moving more stuff under that semaphore. | |
| 19:37:16 | efried | specifically driver-specific claim stuff | |
| 19:37:21 | efried | Not sure if you were around for those conversations. | |
| 19:37:33 | cdent | not that I'm aware of | |
| 19:38:33 | cdent | it's a problem for high throughput nova-computes (like in vmware where the nova-compute is a chokepoint, rather than a member of a nice bit of parallelism like in a big kvm cloud) | |
| 19:40:46 | efried | cdent: well, the plan is to start delegating virt-specific claimage to the virt driver itself. I could see where vmware could take its own semaphore on a specific internal nodeything and background the real claim job, so the next one could come in and get started (but on a different nodeything). | |
| 19:40:53 | efried | and it would be none of RT's business. | |
| 19:41:52 | efried | the more we move claimables into placement, the more we can delegate the claim logic for same to the virt driver. | |
| 19:42:05 | efried | so e.g. PCI devices would eventually become virt driver business. | |
| 19:42:23 | efried | and - as we were discussing this morning - not even stored in the db anymore. | |
| 19:44:05 | cdent | well that would be lovely | |
| 19:57:18 | slaweq | hi, is there any n-meta-api service expert here? Can You take a look at https://bugs.launchpad.net/neutron/+bug/1836642 from Nova PoV? Thx in advance :) | |
| 19:57:18 | openstack | Launchpad bug 1836642 in OpenStack Compute (nova) "Metadata responses are very slow sometimes" [Undecided,New] | |
| 20:01:06 | efried | artom: catching up, are you filing an elastic-recheck profile for bug 1836595 ? | |
| 20:01:06 | openstack | bug 1836595 in neutron "test_server_connectivity_cold_migration_revert failing" [Undecided,New] https://launchpad.net/bugs/1836595 | |
| 20:02:10 | artom | efried, I could - should I be? :) | |
| 20:02:48 | artom | efried, I'm kinda hoping https://review.opendev.org/#/c/670848/ would merge though | |
| 20:03:42 | artom | Speaking of which, gmann ^^ :) | |
| 20:03:51 | artom | (If you're around, not sure about your TZ) | |
| 20:05:11 | efried | artom: This is interesting, the failure on that skip patch http://logs.openstack.org/48/670848/1/check/neutron-tempest-dvr/ed2b81c/testr_results.html.gz | |
| 20:06:19 | artom | efried, looks like feral packets to me | |
| 20:06:26 | efried | conflict deleting allocations, looks like it's tempest itself that's being a placement client. Without having looked at the code, one wonders whether it should be retrying | |
| 20:06:52 | efried | feral packets? how so? | |
| 20:06:53 | artom | Yeah, tempest makes a point not to use any Python clients | |
| 20:07:09 | artom | efried, heh, a lazy way of saying "something unrelated I can't be arsed to debug" :P | |
| 20:07:22 | artom | So it'll do the GET requests itself | |
| 20:07:27 | efried | there is no placement client, my point is that this isn't tempest calling something in nova that's talking to placement and getting a conflict. | |
| 20:07:39 | efried | this is tempest talking directly to placement | |
| 20:07:49 | artom | And that's bad? | |
| 20:08:14 | efried | it means there's likely an opportunity to harden the tempest code so this failure doesn't happen anymore. | |
| 20:08:59 | artom | *facepalm*, oh just got what you're saying | |
| 20:09:11 | artom | Tempest shouldn't be deleting allocations itself, Nova should be doing it | |
| 20:09:47 | artom | I think that response is from Nova though | |
| 20:11:29 | efried | mmm, yes, it looks like you're right. | |
| 20:11:54 | artom | That test is really confusing | |
| 20:13:16 | efried | if the error is coming from nova, it means we have a bug in nova. | |
| 20:16:08 | artom | Well, looks like tempest is creating a server, not waiting for it to become ACTIVE, then immediately deleting it | |
| 20:16:33 | artom | So I suspect that makes Nova hit a race between the build process and the delete request | |
| 20:16:49 | cdent | artom: that sounds right | |
| 20:17:00 | artom | Short term that can be "fixed" by making Tempest wait | |
| 20:17:22 | artom | Longer term I guess we'll need to stick a lock in Nova somewhere | |
| 20:18:49 | cdent | placement log entries are near http://logs.openstack.org/48/670848/1/check/neutron-tempest-dvr/ed2b81c/controller/logs/screen-placement-api.txt.gz#_Jul_15_17_27_35_283720 | |
| 20:20:03 | cdent | handling for the req id starts here http://logs.openstack.org/48/670848/1/check/neutron-tempest-dvr/ed2b81c/controller/logs/screen-n-cpu.txt.gz#_Jul_15_17_27_33_968264 | |
| 20:20:18 | artom | Actually from the little I know about placement it should handle those kinds of races with the generation thing, right? | |
| 20:20:27 | artom | So maybe Nova just needs to handle Placement error better | |
| 20:20:28 | efried | the generation thing is exactly what's happening here | |
| 20:20:55 | efried | alloc deletion was specifically set up so that you couldn't race deletion with some other consumer op | |
| 20:21:24 | efried | so this guard is doing exactly what it's supposed to, and it's the overarching operation that's broken. As you say, trying to delete while creating. | |
| 20:21:30 | cdent | "instance disappeared during build" http://logs.openstack.org/48/670848/1/check/neutron-tempest-dvr/ed2b81c/controller/logs/screen-n-cpu.txt.gz#_Jul_15_17_27_34_806925 | |
| 20:21:49 | efried | only question is where to fix it. | |
| 20:22:24 | efried | it would be possible to redrive the alloc delete until it works. But that's still potentially racy if the other op is trying to create the alloc at the same time. | |
| 20:23:24 | efried | there's a couple races here actually. | |
| 20:23:29 | artom | @instance_state(ACTIVE) for delete :D | |
| 20:23:46 | efried | The other one is if the deletion goes through before the allocation is even created | |
| 20:25:12 | efried | it will not raise an exception (it returns False, but the caller doesn't check that) so we'll end up with a leaked allocation | |
| 20:26:15 | cdent | efried, artom if you end up creating a bug about this, please let me know what it is so I can follow along, tomorrow | |
| 20:27:29 | artom | efried, mind doing it? You have a better grasp of that stuff, and I'm off in 30 minutes anyways, daycare duty | |
| 20:27:46 | efried | ack | |
| 20:48:57 | artom | *snerk* | |
| 20:49:00 | artom | https://bugs.launchpad.net/nova/+bug/1836204 | |
| 20:49:00 | openstack | Launchpad bug 1836204 in OpenStack Compute (nova) "The allocation of VGPU has race problem" [High,Triaged] - Assigned to Alex Xu (xuhj) | |
| 20:49:08 | artom | I guess it hates blacks | |
| 20:49:47 | artom | *shakes head* I'm really sorry | |
| 22:57:32 | efried | dansmith: Do you feel we need a bp/spec for claim_for_instance? | |
| 22:58:09 | efried | there should be no API, db, object, conf, upgrade, or doc impacts | |
| #openstack-nova - 2019-07-16 | |||
| 00:27:04 | openstackgerrit | Merged openstack/nova master: doc: Add links to novaclient contributor guide https://review.opendev.org/669028 | |
| 00:46:31 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Make the '--config-drive' be a boolean option https://review.opendev.org/653683 | |
| 02:23:22 | openstackgerrit | Ming Jin proposed openstack/nova master: Nova handle empty vcpu_pin_set scenario https://review.opendev.org/670935 | |
| 02:25:59 | openstackgerrit | Yongli He proposed openstack/python-novaclient master: Microversion 2.75 - show server topology https://review.opendev.org/670790 | |
| 03:05:44 | openstackgerrit | Ming Jin proposed openstack/nova master: Nova handle empty vcpu_pin_set scenario https://review.opendev.org/670935 | |
| 04:46:47 | openstackgerrit | Yongli He proposed openstack/python-novaclient master: Microversion 2.75 - show server topology https://review.opendev.org/670790 | |
| 05:09:31 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix a unit test for pre_live_migration method https://review.opendev.org/670948 | |
| 06:27:21 | openstackgerrit | Alex Xu proposed openstack/nova master: Add the virt driver interface for claim and unclaim the devices https://review.opendev.org/670782 | |
| 06:27:21 | openstackgerrit | Alex Xu proposed openstack/nova master: Move the allocation retrieving early https://review.opendev.org/670783 | |
| 06:27:22 | openstackgerrit | Alex Xu proposed openstack/nova master: Claim and unclaim deivces when spawn and destory the instance https://review.opendev.org/670784 | |
| 06:27:22 | openstackgerrit | Alex Xu proposed openstack/nova master: Claim/unclaim the deivces when rebuilding the instance https://review.opendev.org/670785 | |
| 06:27:23 | openstackgerrit | Alex Xu proposed openstack/nova master: Claim/unclaim the devices when resizing instance https://review.opendev.org/670786 | |
| 06:27:23 | openstackgerrit | Alex Xu proposed openstack/nova master: Populate the existing mediated devices in the libvirt device manager https://review.opendev.org/670787 | |
| 06:38:54 | openstackgerrit | Yongli He proposed openstack/nova master: clean up orphan instances https://review.opendev.org/627765 | |
| 07:06:34 | openstackgerrit | melanie witt proposed openstack/nova master: nova-manage db archive_deleted_rows is not multi-cell aware https://review.opendev.org/507486 | |
| 07:51:39 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/stein: Add 'path' query parameter to console access url https://review.opendev.org/670972 | |
| 08:28:20 | tssurya | takashin: are you around ? | |
| 08:29:33 | takashin | tssurya: yes. | |
| 08:29:43 | tssurya | you review comment here: https://review.opendev.org/#/c/666792/2/novaclient/tests/unit/v2/test_shell.py@4277 | |
| 08:29:51 | tssurya | can I ask why 74 was added there in the first place ? | |
| 08:30:13 | tssurya | could you respond to the question when you have time ? thanks in advance | |
| 08:30:27 | tssurya | your* | |
| 08:30:41 | takashin | tssurya: okay | |
| 08:38:57 | tssurya | @cores: reviews for https://review.opendev.org/#/c/645611/ would be appreciated, its in the runways. Thanks in advance | |
| 09:13:52 | openstackgerrit | Surya Seetharaman proposed openstack/python-novaclient master: API microversion 2.75: Add 'power-update' external event https://review.opendev.org/666792 | |
| 09:25:26 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: doc: Clarify versioned wrapped method https://review.opendev.org/670989 | |
| 09:37:04 | slaweq | sean-k-mooney: hi, can You help me with one bug? | |