| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-19 | |||
| 13:41:50 | bauzas | mriedem: around ? | |
| 13:42:20 | bauzas | mriedem: I saw your comment on https://review.opendev.org/#/c/670112/ | |
| 13:42:25 | bauzas | mriedem: thanks for it | |
| 13:42:31 | bauzas | that said, I have one concern | |
| 13:43:00 | bauzas | if we are adding some check that depends on an unfix bug, then we would need to fix this test when the bugfix arrives | |
| 13:43:30 | bauzas | I mean, I'm totally ok with doing an evacuation | |
| 13:44:19 | bauzas | but I just want to make sure that we would still have orphaned allocations even if someone fixes the "undeleted allocations" stuff when a host is deleted | |
| 13:44:20 | mriedem | there are other ways to orphan migration-based allocations, like resizing a server and then before confirming delete the source compute service | |
| 13:44:42 | mriedem | there might be other ways | |
| 13:44:48 | mriedem | i do'nt know when those bugs will be fixed | |
| 13:44:54 | mriedem | my brain is also not really functioning atm | |
| 13:44:58 | bauzas | no worries | |
| 13:45:19 | bauzas | you know what ? I'll just stick with your plan and leave a big fat comment, others could voice | |
| 13:57:03 | efried | Nova meeting in 3 minutens | |
| 13:57:15 | efried | ^ pseudo-german, apparently | |
| 13:59:23 | mriedem | efried: so you're running it? | |
| 13:59:26 | mriedem | i just updated the agenda fwiw | |
| 13:59:39 | efried | oh, thank you, I guess I'll get a merge conflict then | |
| 14:12:10 | bauzas | oh crap, discovered a very interesting case | |
| 14:12:18 | bauzas | https://docs.openstack.org/api-ref/placement/?expanded=delete-allocations-detail#delete-allocations deletes all allocations from this consumer | |
| 14:12:37 | bauzas | but that supposes there are no duplicate allocations having the same consumer UUID between two RPs | |
| 14:12:51 | bauzas | which isn't the case when you evacuate an instance and delete the source host | |
| 14:13:16 | bauzas | we're missing some API syntax to say 'I just want *this* allocation attached to *this* RP | |
| 14:16:23 | bauzas | that's sad because if you run my new audit command, then you'll loose your allocation on an existing instance until you heal it | |
| 14:18:38 | gibi | bauzas: In my mind there is not two allocation with the same consumer UUID but there is a singel consumer that allocates from two RPs. | |
| 14:18:56 | bauzas | gibi: sure, but at the end, we delete both records | |
| 14:19:21 | bauzas | anyway, I'll amend my audit command to be preventive and claim there are duplicate records | |
| 14:19:28 | bauzas | instead of deleting | |
| 14:19:31 | stephenfin | mriedem: fwiw, that live-migrate job failed because placement-api didn't start in time. I'm assuming that couldn't be related to the change, which adds checks on compute-manager startup | |
| 14:19:45 | mriedem | stephenfin: yeah i just left a comment about that | |
| 14:19:47 | mriedem | likely a slow node | |
| 14:19:52 | mriedem | devstack times out at 60 seconds | |
| 14:20:17 | stephenfin | Ah, cool. Thanks for checking | |
| 14:28:09 | mriedem | stephenfin: fwiw it looks rare: | |
| 14:28:12 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22%5BERROR%5D%5C%22%20AND%20message%3A%5C%22placement-api%20did%20not%20start%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d | |
| 14:33:53 | donnyd | FWIW all three of those ran on FN, so I am curious if its slowness or something else | |
| 14:36:34 | kashyap | mriedem: Dear cupid, just to bounce back on the ping from yesterday, I've responded earlier in the day to mnaser -- http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-09-19.log.html#t2019-09-19T08:25:57 | |
| 14:37:03 | kashyap | mriedem: And we ended up being on the same sentence, same word: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-09-19.log.html#t2019-09-19T11:21:12 | |
| 14:37:07 | kashyap | :D | |
| 14:37:33 | mriedem | donnyd: i'm assuming slowness, placement-api workers took 70+ seconds to start, | |
| 14:37:36 | mriedem | devstack times out at 60 | |
| 14:38:05 | mriedem | kashyap: ack | |
| 14:39:06 | kashyap | I'll also send a patch to libvirt docs to explicitly note that 'host-model' will be "expanded" for the live guest. | |
| 14:41:01 | donnyd | https://grafana.fortnebula.com/d/9MMqh8HWk/openstack-utilization?orgId=2&from=1568848907005&to=1568854899362 | |
| 14:41:25 | donnyd | Here is the load snapshot from one of the failures.. doesn't look like anything is overloaded | |
| 14:47:02 | lyarwood | does anyone recall where we actually call initialize_connection during a LM for the destination when using cinderv2? | |
| 14:47:21 | lyarwood | I was sure it was hidden in pre_live_migration but I can't seem to find it. | |
| 14:53:02 | openstackgerrit | sean mooney proposed openstack/nova-specs master: Add spec for vm scoped sriov numa affinity https://review.opendev.org/683174 | |
| 14:53:26 | sean-k-mooney | efried: so you are saying i should start writing more specs now :) | |
| 14:54:12 | efried | sean-k-mooney: no, I'm saying there's no point writing a spec for something brand new. | |
| 14:54:21 | sean-k-mooney | https://review.opendev.org/#/c/674072/3 is the code for ^ by the way | |
| 14:54:41 | sean-k-mooney | because we have enough of a backlog already to finish | |
| 14:54:57 | mriedem | lyarwood: it is in pre_live_migration from the compute manager | |
| 14:55:09 | mriedem | eventually calls down into nova.virt.block_device - refresh_connection_info method | |
| 14:55:25 | mriedem | lyarwood: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6719 | |
| 14:55:47 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/block_device.py#L677 | |
| 14:55:56 | efried | sean-k-mooney: that, and because it's absolute hell for the "customer" relying on an upstream feature to meet a business commitment to have no better than a 60% chance of the feature actually landing, despite being "approved" (or even *prioritized*) for the release. I'm hoping by reducing scope we can increase that percentage. | |
| 14:56:10 | lyarwood | mriedem: thanks! | |
| 14:56:51 | mriedem | lyarwood: it's probably worth putting a comment by that call in the compute manager saying for legacy attached BDMs that will update the bdm connection_info using the dest host connector | |
| 14:57:17 | lyarwood | mriedem: ack, I'll do that shortly this isn't the first time I've missed this. | |
| 15:19:47 | mriedem | lyarwood: remember, anything involving block devices has to be 3 layers of abstraction removed from where you'd expect it to be happening | |
| 15:38:00 | donnyd | mriedem: 683166 in neutron is so the def route in ipv6 providers isn't purged when the public side of the network is created in the job | |
| 15:40:10 | mriedem | ok. do we have job failures as a result of that? if so, do you have a link to one so we can try to categorize it in elastic-recheck? | |
| 15:42:00 | openstackgerrit | Takashi Kajinami proposed openstack/nova master: conf: Add three new '[libvirt] live_migration_*' options https://review.opendev.org/456571 | |
| 15:42:20 | mriedem | melwitt: on https://review.opendev.org/#/c/541420/ i'll try removing the weird dict/attribute stuff since that might be fixed now since i originally posted that patch, | |
| 15:42:28 | mriedem | melwitt: but i wasn't sure what you were saying with "But, I was wondering, should we cover the detach case in the DNM patch above this one too? (I left a comment there)." | |
| 15:42:41 | mriedem | oh nvm let me look at your comment first | |
| 15:42:54 | openstackgerrit | Takashi Kajinami proposed openstack/nova master: conf: Add three new '[libvirt] live_migration_*' options https://review.opendev.org/456571 | |
| 15:43:22 | donnyd | mriedem: 2019-09-19 09:50:54.608322 | controller | +++ lib/neutron_plugins/services/l3:_neutron_create_public_subnet_v6:330 : oscwrap --os-cloud devstack-admin --os-region RegionOne subnet create --ip-version 6 --gateway 2001:db8::2 --network 0dd00e84-7a3a-41a5-bcf0-79097e5c1d1f --subnet-range 2001:db8::/64 --no-dhcp ipv6-public-subnet | |
| 15:43:22 | donnyd | https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_650/682647/3/check/ironic-standalone-ipa-src/65091a8/job-output.txt | |
| 15:43:42 | donnyd | there are a couple more examples | |
| 15:44:27 | donnyd | https://www.irccloud.com/pastebin/7aez1E6y/ | |
| 15:46:17 | melwitt | mriedem: ok cool. I never knew there was any issue with attr going to the object | |
| 15:46:29 | mriedem | donnyd: sorry i'm not sure what i'm looking for here that indicates the job fails because of that | |
| 15:47:02 | donnyd | The gist is on v6 providers the def route is dropped at this step and it may or may not come back up in time depending on RA / fw connections | |
| 15:48:28 | mriedem | donnyd: as indicated by "[Zuul] Log Stream did not terminate" ? | |
| 15:48:34 | donnyd | frickler: may have more insights | |
| 15:49:04 | mriedem | "[Zuul] Log Stream did not terminate" shows up in passing jobs so that's not related | |
| 15:49:36 | mriedem | anyway, i don't have +2 on devstack anyway, i was just doing a drive by review | |
| 15:50:59 | donnyd | mriedem: well I was hoping for at least 6 more +1's | |
| 15:52:50 | melwitt | 6 specifically? lol | |
| 15:52:58 | donnyd | yes | |
| 15:53:07 | melwitt | :) | |
| 15:53:29 | donnyd | but no more than 8... that is too many | |
| 15:53:50 | mriedem | back in the icehouse days i think 6 +1s from a single vendor equaled one +2 | |
| 15:54:20 | mriedem | especially all within 1 hour of each other after their scrum call ended | |
| 15:54:31 | melwitt | lol | |
| 15:54:46 | melwitt | too real | |
| 15:55:06 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: Stop using volume_api.initialize/terminate_connection https://review.opendev.org/682508 | |
| 15:55:09 | donnyd | mriedem: I am really just working on upping my level of irc sarcasm - ircasm | |
| 15:55:28 | mriedem | donnyd: i should probably use o-) or :P or something until you're used to it | |
| 15:55:45 | donnyd | :( | |
| 15:56:51 | donnyd | ok | |
| 15:57:56 | mriedem | somehow i've offended you and i'm not sure how i did | |
| 15:58:05 | mriedem | i meant, i'm 80% sarcasm | |
| 15:58:08 | mriedem | so don't take me literally | |
| 16:09:38 | mriedem | https://review.opendev.org/#/c/679640/ failed with something i haven't seen in the gate before | |
| 16:09:46 | mriedem | https://zuul.opendev.org/t/openstack/build/998bcd66b810467fae1664aaf280d53d/log/controller/logs/screen-n-cpu.txt.gz?severity=4#25797 | |
| 16:09:52 | mriedem | os_brick.exception.VolumeDeviceNotFound: Volume device not found at /dev/disk/by-id. | |
| 16:10:57 | mriedem | lyarwood: you might care about that ^ since it's a luks encrypted volume test | |