| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-25 | |||
| 21:22:50 | efried | thx | |
| 21:23:45 | mriedem | zzzeek: if you're about, i have a test which keeps failing trying to insert a resource to the db with this: | |
| 21:23:46 | mriedem | sqlalchemy.exc.InvalidRequestError: This session is in 'inactive' state, due to the SQL transaction being rolled back; no further SQL can be emitted within this transaction. | |
| 21:24:01 | mriedem | it's using a transaction factory context manager thing from oslo.db, | |
| 21:24:05 | zzzeek | mriedem: that woudl suggest someting went wrong prior to the thing you are trying t do | |
| 21:24:23 | zzzeek | like, youre in an exception handler or something like that | |
| 21:24:24 | mriedem | i would think if there was a problem with the insert, like a unique constraint failure or something, it'd be logged but i'm not seeing that | |
| 21:24:30 | mriedem | ah... | |
| 21:24:38 | mriedem | yes there is one of those in play | |
| 21:24:59 | zzzeek | mriedem: so, say you're in an exception throw, then you try to do somethign with an object, and something on it isn't loaded | |
| 21:25:03 | zzzeek | mriedem: that would cause this | |
| 21:25:24 | zzzeek | err inside a catch, that is | |
| 21:26:14 | mriedem | yeah it's blowing up from the transaction context manager added here: https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L691 | |
| 21:26:25 | mriedem | and likely from something in https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L703 | |
| 21:26:56 | mriedem | the weird thing is i'm hitting this https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L716 | |
| 21:26:59 | mriedem | so it shouldn't be re-raising | |
| 21:29:44 | mriedem | oh maybe this is failing https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L745 | |
| 21:29:50 | mriedem | adding some try/except/log | |
| 21:30:20 | mriedem | nope not hitting that | |
| 21:30:33 | mriedem | i'm definitely getting here https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L716 | |
| 21:39:34 | openstackgerrit | Eric Fried proposed openstack/nova master: Conf option cross-reference in config docs https://review.opendev.org/583025 | |
| 21:41:12 | zzzeek | mriedem: these are hard to debug | |
| 21:41:50 | mriedem | so we have another test that does this same thing but only within 1 nova-compute service, so i'm wondering if it's something with the thread local context stuff in oslo.db | |
| 21:41:55 | mriedem | b/c this test has 2 services using the same context | |
| 21:42:30 | mriedem | the other one was https://review.opendev.org/#/c/675496/ and things worked like a charm there | |
| 21:42:48 | mriedem | we hit DBDuplicateEntry, then do a get-and-update and squash the re-raise | |
| 21:42:52 | mriedem | but that's not working this time | |
| 21:47:38 | efried | dustinc: It looks like the ironic CI is failing again on service discovery, but I'm not sure why... | |
| 21:51:29 | efried | maybe not discovery | |
| 22:09:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Clear RT.compute_nodes entry if re-balance update fails https://review.opendev.org/684849 | |
| 23:10:57 | openstackgerrit | Merged openstack/nova master: Ignore sqla-migrate inspect.getargspec deprecation warnings on py36 https://review.opendev.org/684781 | |
| 23:11:04 | openstackgerrit | Merged openstack/nova master: Ignore warning from sqlalchemy-migrate https://review.opendev.org/684772 | |
| 23:11:10 | openstackgerrit | Merged openstack/nova master: docs: Clarify everything CPU pinning https://review.opendev.org/683437 | |
| 23:11:17 | openstackgerrit | Merged openstack/nova master: docs: Update CPU topologies guide to reflect the new PCPU world https://review.opendev.org/683485 | |
| 23:11:28 | openstackgerrit | Merged openstack/nova master: Move HostNameWeigher to a common fixture https://review.opendev.org/683874 | |
| 23:11:39 | openstackgerrit | Merged openstack/nova master: Correct link to placement upgrade notes https://review.opendev.org/684661 | |
| #openstack-nova - 2019-09-26 | |||
| 02:12:05 | openstackgerrit | Brin Zhang proposed openstack/nova master: Filter migrations by user_id/project_id https://review.opendev.org/674243 | |
| 02:51:35 | LiangFang | hi, could anybody share how to add agenda for next weekly meeting? Is there any etherpad link? thanks | |
| 05:42:21 | gibi | LiangFang: add your item to the agend be editing the wiki https://wiki.openstack.org/wiki/Meetings/Nova | |
| 06:27:56 | openstackgerrit | zhurong proposed openstack/nova master: [WIP] add cyborg tempest job https://review.opendev.org/670999 | |
| 08:13:41 | kashyap | ozzzo: Hi, just saw a few mentions in the scrollback; /me goes to look for the salient bits | |
| 08:14:52 | kashyap | ozzzo: Okay, saw your question: if you're using (a) 'host-passthrough'; and (b) mismatched microcode b/n Compute nodes, then no, you will _not_ be able migrate an instance across the said nodes. | |
| 08:16:54 | kashyap | ozzzo: In my talk, I did mention the impact of microcode on live migration, when using 'host-passthrough': | |
| 08:17:08 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reset forced_destination before migration at a proper time https://review.opendev.org/684546 | |
| 08:17:08 | kashyap | ozzzo: efried: See slide-18: https://kashyapc.fedorapeople.org/Effective-Virtual-CPU-Configuration-in-Nova-Berlin2018.pdf | |
| 08:17:08 | openstack | bug 1845291 in OpenStack Compute (nova) "migration is not recheduled if the server originally booted with --availability-zone |
|
| 08:17:08 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Functional reproduction for bug 1845291 https://review.opendev.org/684545 | |
| 08:18:18 | kashyap | ozzzo: To quote at note at the end of slide-18 (which talks about 'host-passthrough': "Along with identical CPUs, identical kernel and microcode are a must for VM live migration!" | |
| 08:20:30 | kashyap | ozzzo: I'd also toss in, "identical BIOS settings" b/n the two COmpute nodes, too. | |
| 09:45:50 | victor286 | hello,so many people! | |
| 09:55:17 | openstackgerrit | Ivaylo Mitev proposed openstack/nova master: Fix incorrect usages of fake moref in VMware tests https://review.opendev.org/685020 | |
| 09:56:29 | kashyap | ozzzo: Also responded on e-mail | |
| 10:00:53 | openstackgerrit | Ivaylo Mitev proposed openstack/nova master: Fix incorrect usages of fake moref in VMware tests https://review.opendev.org/685020 | |
| 11:27:55 | openstackgerrit | Merged openstack/nova master: doc: attaching virtual persistent memory to guests https://review.opendev.org/680300 | |
| 11:28:02 | openstackgerrit | Merged openstack/nova master: Add a prelude for the Train release https://review.opendev.org/683327 | |
| 11:41:14 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: [WIP] add cyborg tempest job https://review.opendev.org/670999 | |
| 11:46:30 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reset forced_destination before migration at a proper time https://review.opendev.org/684546 | |
| 11:46:30 | openstack | bug 1845291 in OpenStack Compute (nova) "migration is not recheduled if the server originally booted with --availability-zone |
|
| 11:46:30 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Functional reproduction for bug 1845291 https://review.opendev.org/684545 | |
| 13:12:40 | openstack | Launchpad bug 1845243 in OpenStack Compute (nova) stein "Nested 'path' query param in console URL breaks serialproxy" [High,Confirmed] | |
| 13:12:40 | mriedem | efried: we might want to consider https://bugs.launchpad.net/nova/+bug/1845243 as train-rc-potential for rc2 since the regression was introduced in train, it was also backported and released on stein and rocy | |
| 13:12:41 | mriedem | *rocky | |
| 13:13:26 | mriedem | so i think we have 2 potential bugs for rc2 https://bugs.launchpad.net/nova/+bugs?field.tag=train-rc-potential but i think we could still release rc1 today | |
| 13:16:48 | mriedem | i've updated the rc1 release patch https://review.opendev.org/#/c/684086/ | |
| 13:17:53 | mriedem | yaawang: were you able to test this fix for the numa live migration bug you reported? https://review.opendev.org/#/c/684409/ | |
| 13:28:21 | kashyap | sean-k-mooney: efried This was my answer to ozzzo, in an "FAQ form" :D - https://kashyapc.fedorapeople.org/CPU-microcode-host-passthrough-and-live-migration.txt | |
| 13:28:41 | kashyap | mriedem: ^ (But you might already know that answer.) | |
| 13:32:24 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Ignore sqla-migrate inspect.getargspec deprecation warnings on py36 https://review.opendev.org/685072 | |
| 13:39:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Clear RT.compute_nodes entry if re-balance update fails https://review.opendev.org/684849 | |
| 13:39:18 | mriedem | kashyap: lgtm, i remember you talking about microcode in your presentation | |
| 13:39:18 | openstack | bug 1841481 in OpenStack Compute (nova) "Race during ironic re-balance corrupts local RT ProviderTree and compute_nodes cache" [Medium,In progress] https://launchpad.net/bugs/1841481 - Assigned to Matt Riedemann (mriedem) | |
| 13:39:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1841481 https://review.opendev.org/684840 | |
| 13:39:38 | mriedem | if a FAQs style thing is useful for that docs page i'm fine with adding one | |
| 13:39:47 | mriedem | we recently did that for the bdms docs | |
| 13:40:01 | mriedem | https://docs.openstack.org/nova/latest/user/block-device-mapping.html#faqs | |
| 13:41:11 | kashyap | Ah, interesting | |
| 13:41:28 | kashyap | mriedem: Yeah, I could actually add some more - expand it to cover the other two modes | |
| 13:41:36 | kashyap | Because, this is going to become a question for more and more | |
| 13:41:56 | kashyap | As they slowly realize and start upgrading microcodes ... | |
| 13:42:58 | kashyap | ... "hey, live migration is broken, what now? :D" | |
| 13:45:40 | kashyap | mriedem: A quick q. on above -- since we're so close up to GA, is changing such defaults frowned upon? (Have a gander at the commit message / release note) | |
| 13:45:51 | efried | mriedem: ack, are we waiting on anything else to cut rc1 today? | |
| 13:47:06 | kashyap | In short: it's a legit issue, also fixed in the kernel. But relaxing the 'strict' default in Nova will definitely avoid surprises like: https://lkml.org/lkml/2018/7/24/152 ("VM boot failure on nodes not having DMA32 zone") | |
| 13:50:47 | mriedem | kashyap: i can't really speak to that change but yeah that's pretty non-trivial for the day of RC1 | |
| 13:50:53 | mriedem | efried: i don't think so | |
| 13:51:20 | efried | ight, I'll go (update and) ack the release patch I guess. | |
| 13:51:23 | mriedem | kashyap: also, "fixed in the kernel" might be something for rhosp but could be years for someone using suse or canonical openstack to pick up i'd think | |
| 13:51:28 | mriedem | efried: i already did | |
| 13:51:35 | mriedem | update htat is | |
| 13:51:37 | mriedem | just needs an ack | |
| 13:51:39 | efried | coo | |
| 13:52:35 | efried | done | |
| 13:52:37 | kashyap | mriedem: I'm not one of those who think "downstream first" ;-) When I say "fixed in the kernel" - it's in 4.19, and many of those said distros will backport it | |
| 13:52:46 | kashyap | It's a super small, isloated change | |
| 13:54:48 | kashyap | mriedem: Okay, ACK - for the RC1. But FWIW, the change is quite _safe_ and self-contained: and makes Nova instances work in more NUMA setups | |
| 13:55:41 | efried | kashyap: If you wanted to be super safe, you could make a [workaround] conf opt, defaulting to "prefer", to allow consumers to revert to "strict" if they want. | |
| 13:56:05 | efried | deprecate it immediately (assuming we don't want to continue to allow it) and remove it in U. | |
| 13:56:44 | kashyap | (Also, to preempt a thought: no, it's not "urgent for RHOS". I just think since Train also has NUMA-aware live migration; this change only helps. I did a crap load background reading on this, to understand the issue) | |
| 13:56:53 | kashyap | efried: Ah, good point on 'workarounds' | |
| 13:57:59 | kashyap | I'll mull on it today; stephenfin - might want to have a look at the above, it's in your wheelhouse - https://review.opendev.org/#/c/684375/ ("libvirt: Use the `preferred` memory allocation mode for NUMA | |