| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-10-28 | |||
| 12:57:36 | gibi | lyarwood: ck | |
| 12:57:40 | gibi | lyarwood: ack | |
| 13:12:22 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: DNM: testing copy image with non admin policy https://review.opendev.org/760128 | |
| 13:46:16 | bauzas | wow, this was violent | |
| 13:46:28 | kashyap | What was violent? | |
| 13:46:28 | bauzas | gibi: my laptop crashed but I'm rejoining | |
| 14:55:52 | lyarwood | zoom-- | |
| 14:56:49 | lyarwood | haha finally get it to work and it's time for a break, typical. | |
| 15:16:20 | melwitt | lyarwood: I saw, the gate is super angry :( | |
| 15:19:43 | lyarwood | melwitt: yup indeed thanks for checking things overnight btw | |
| 15:19:49 | lyarwood | melwitt: it's merged now finally | |
| 15:21:37 | melwitt | lyarwood: yeah, I saw it got force merged if I understood correctly. and yeah, want to get a handle on the gate bugs and figure out some root causes. things have been crazy lately | |
| 15:56:53 | lyarwood | bauzas: mute :) | |
| 15:57:13 | bauzas | shit | |
| 15:57:22 | lyarwood | np | |
| 15:57:38 | bauzas | if anyone knows whether zoom supports space tabs for muting/unmuting automatically, I'm your man | |
| 16:11:39 | bauzas | F32-- | |
| 16:11:46 | bauzas | killed my zoom and a couple of apps | |
| 16:26:00 | gibi | #nova now Deprecate os-hypervisors API | |
| 16:28:58 | openstackgerrit | Merged openstack/nova master: Add placeholder migrations for Victoria backports https://review.opendev.org/754396 | |
| 16:44:10 | bauzas | gibi: wrong channel ^ ;) | |
| 16:44:44 | gibi | :D | |
| 16:44:50 | melwitt | gibi, elod, lyarwood: been meaning to ask for awhile ... would appreciate your opinion about these two open stable-only changes whether you think it's appropriate given the oldness of queens https://review.opendev.org/#/q/topic:bug/1731668 | |
| 16:45:02 | melwitt | tl;dr is there was a bug in some transitionary code that is triggered by lots of parallel requests to placement (example: running Rally). impact (if I read the code correctly) is instances go into ERROR state for failure to create placement allocations | |
| 16:45:17 | gibi | melwitt: I have to get back to these tomorrow morning | |
| 16:45:27 | melwitt | np thanks | |
| 16:50:29 | elod | melwitt: sure, I'll try to understand the whole case and figure out whether it fits to be backported or not. but I'm a bit lagging behind myself, so I won't promise anything right now o:) | |
| 16:50:43 | elod | and thanks for the tl;dr :] | |
| 16:52:02 | melwitt | elod: np. yeah it's just a bug, if you try to create a bunch of instances in parallel in queens, you can hit this bug and cause lots of instances to fall into ERROR state as they fail during the scheduling step | |
| 16:52:13 | lyarwood | melwitt: I'm slightly confused, why is this stable-only? | |
| 16:52:29 | melwitt | lyarwood: bug is in transitionary code that no longer exists | |
| 16:52:41 | lyarwood | melwitt: ah sorry didn't gork that correctly in your tldr | |
| 16:53:12 | lyarwood | melwitt: and just to be clear, it no longer exists in Placement right? | |
| 16:53:16 | lyarwood | melwitt: not just Nova | |
| 16:53:24 | melwitt | lyarwood: np, I'm sure I didn't explain it well. also, note the func test needs real MySQL to repro the bug because it has to do with database transaction isolation which sqlite has none of | |
| 16:53:37 | melwitt | lyarwood: correct, no longer in placement | |
| 16:53:45 | lyarwood | melwitt: ack thanks | |
| 17:12:06 | lyarwood | melwitt: so the change LGTM as a stable only change but I'd like other master cores with placement/db exp to take a look before we merge this | |
| 17:12:20 | lyarwood | melwitt: could you also add the context about why this is stable only in the commit message itself? | |
| 17:12:30 | lyarwood | actually both of them as they are both marked as stable only | |
| 17:20:32 | openstackgerrit | Merged openstack/nova stable/victoria: libvirt: 'video.vram' property must be an integer https://review.opendev.org/757616 | |
| 17:42:02 | melwitt | lyarwood: I tried to do that as "As part of data migration code" but I acknowledge that's not good enough | |
| 18:01:45 | mnaser | hmm, it looks like live migration when using routed networks can result in the scheduler picking the wrong hosts (systems that are on a different segment?) | |
| 18:01:55 | mnaser | am I missing something or this seems to be something that's not yet implemetned | |
| 18:03:26 | lyarwood | mnaser: https://review.opendev.org/#/q/topic:bp/routed-networks-scheduling - I think that's still a WIP | |
| 18:13:07 | rm_work | re: the way nova migrations / DB stuff is handled -- if the DB is upgraded from say, rocky->stein, and then we try to run stein but have some issue... could the rocky service run on the stein DB schema? or would it explode violently? or maybe more subtly break? | |
| 18:16:31 | sean-k-mooney | mnaser: ya as lyarwood said that is not implemented nova currenly has not awareness of segments | |
| 18:17:01 | sean-k-mooney | mnaser: bauzas will be working on it for wallaby | |
| 18:22:37 | melwitt | rm_work: yes it can, this is what happens during a rolling upgrade from N to N+1, see details https://docs.openstack.org/nova/latest/user/upgrade.html | |
| 18:23:31 | rm_work | ohh | |
| 18:23:35 | rm_work | "These DB schema changes are done in a way that both the N and N+1 release can perform operations against the same schema." | |
| 18:23:37 | rm_work | A++++ | |
| 18:23:40 | rm_work | thanks! | |
| 18:24:43 | melwitt | yw | |
| 18:28:21 | rm_work | followup -- placement is microversioned, right? so ... we could feasibly run nova at Stein and placement at ... Master? :P | |
| 18:28:52 | rm_work | (once we've done the initial migration to standalone placement service/DB, in stein) | |
| 18:30:35 | rm_work | there is only a dependency on the NOVA side that placement must be at least at the same version as nova | |
| 18:30:54 | rm_work | there's no such reverse dependency on the placement side? | |
| 18:31:07 | sean-k-mooney | rm_work: correct | |
| 18:31:18 | melwitt | yeah. as mentioned in the doc, placement must be same or ahead of nova bc of the dependency but other than that, it is designed to work independently by way of the API microversions | |
| 18:31:20 | sean-k-mooney | nova has a min placment version it works with | |
| 18:31:22 | rm_work | our cloud is about to look like: Nova(Stein), Placement(Master), Neutron(Train), Octavia(Master) | |
| 18:31:27 | sean-k-mooney | but placement does not have any depency on nova | |
| 18:31:57 | sean-k-mooney | rm_work: that should be fine at least Nova(Stein), Placement(Master), Neutron(Train) | |
| 18:31:58 | rm_work | any specific neutron version dependencies? | |
| 18:32:16 | rm_work | If we jump Nova to Victory with Neutron still at Train, could we see problems? | |
| 18:32:26 | sean-k-mooney | it depends on the feature you are using but we disable feature if neutron is not new enough | |
| 18:32:34 | rm_work | ok, so it handles it | |
| 18:32:57 | sean-k-mooney | rm_work: that is not tested. in thory it might be ok generally you want to upgrade neutorn before nova | |
| 18:33:01 | rm_work | assuming that'd only be for newer features anyway, and if we're JUST upgrading nova, we wouldn't already be relying on them | |
| 18:33:06 | rm_work | ok, noted | |
| 18:33:07 | sean-k-mooney | but it shoudl work for those versions | |
| 18:33:25 | rm_work | it's probably fine to plan to "be safe" and just upgrade neutron first, not really a huge issue | |
| 18:33:46 | rm_work | how common do you think it is for folks to run clouds with services at completely different versions? | |
| 18:33:54 | sean-k-mooney | https://github.com/openstack/grenade is what we use for upgrade testing in the ci | |
| 18:34:09 | sean-k-mooney | so usign the same or a similar orderign is likely to work better | |
| 18:34:55 | rm_work | I haven't seen it discussed a ton, but I've been giving a lot of thought to how Openstack at this point really shouldn't have a ton of requirements as far as versions go, in many cases, so running a cloud with patchwork versioning seems actually quite reasonable | |
| 18:35:07 | rm_work | so we've moved in that direction | |
| 18:35:36 | rm_work | starting to wonder if that might be worth a presentation next summit ;) | |
| 18:35:55 | sean-k-mooney | it might | |
| 18:36:18 | sean-k-mooney | this summerises how upgrades should work in openstack https://github.com/openstack/grenade#theory-of-upgrade | |
| 18:36:51 | sean-k-mooney | thats specific to singel version upgrades | |
| 18:36:54 | rm_work | cool thanks, I THINK i've gone through this but it's been a while so will give it a re-read | |
| 18:37:10 | rm_work | yeah, seems no matter what we should probably avoid fast-forward / skip-level | |
| 18:37:32 | sean-k-mooney | rm_work: this is the order of upgrades done by grenade https://github.com/openstack/grenade/tree/master/projects | |
| 18:38:04 | sean-k-mooney | i acutlly expected cinder to go before nova but i guess it does not | |
| 18:38:23 | rm_work | I have been studying https://docs.openstack.org/placement/latest/admin/upgrade-to-stein.html EXTENSIVELY, heh | |
| 18:38:27 | rm_work | that's our current challenge | |
| 18:38:52 | sean-k-mooney | its more a case of manageing dependices. | |
| 18:39:01 | sean-k-mooney | keystone has none outside oslo | |
| 18:39:09 | melwitt | +1 running services at different versions is reasonable. it's designed to work that way. there are likely limitations (like placement needs to be >= nova) but prior to that things were supposed to work with mixed versions | |
| 18:39:09 | sean-k-mooney | placemnt only depneds on keystone | |
| 18:39:21 | sean-k-mooney | ectra | |
| 18:39:28 | sean-k-mooney | rm_work: are you using a containerised install | |
| 18:39:34 | sean-k-mooney | or package install | |
| 18:39:35 | rm_work | mixed :P | |
| 18:39:45 | rm_work | we create our own portable venvs for most services | |
| 18:39:52 | rm_work | but placement will be our first actual container | |
| 18:39:54 | sean-k-mooney | ah perfect | |
| 18:40:16 | melwitt | fast-forward is ok but skip-level is not, you could get screwed if data migration code was removed in the middle of what you're skipping | |