Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-05-30
16:41:19 geguileo dansmith: yes, that could happen
16:42:05 bauzas could nova ask brick such thing ?
16:42:08 dansmith geguileo: maybe everywhere we currently do connect_volume() we do a disconnect first and ignore an error? that generates a lot of churn, but maybe that's safer?
16:42:29 dansmith bauzas: that's what I was wondering.. if there was some validate_volume() call or something we could run to know if we should run connect
16:42:41 bauzas like, before doing connect_volume(), do a check_vol()
16:42:43 bauzas ahah
16:42:54 geguileo dansmith: os-brick will be doing that already with optimized code
16:43:20 geguileo dansmith: that's the issue that I'm bringing, that the new code that cleans things up break the idempotency, so there's no need for nova to do that call you mention
16:43:37 geguileo os-brick cannot do a proper check_vol like we would want
16:43:48 geguileo because the volume can have changed in the background
16:44:06 geguileo for example change the size, point a different volume in the backend (like happened in the CVE)
16:44:26 dansmith okay, I'm just not sure how we can know what the right thing to do is
16:44:27 geguileo there are a bunch of messes that could happen, and I'm not sure we would want this to depend on nova calling a method
16:45:04 geguileo dansmith: well, if you call connect_volume all instances in Nova should use the device it returns from that moment onwards
16:45:10 bauzas geguileo: I'm confused, who's responsible for knowing that all connectors are present on the host ?
16:45:17 bauzas I thought it was brick
16:45:23 dansmith geguileo: okay I thought you didn't want us to do that
16:45:27 geguileo what do you mean by connectors?
16:45:48 bauzas wrong wording, my bad.
16:45:51 dansmith geguileo: or are you just saying any time we call connect_volume() we need to examine the result and make sure the instance is (changed, if necessary) to use that/
16:46:16 bauzas geguileo: I'd say the device number
16:46:44 geguileo dansmith: what I don't want is nova to call connect_volume, get the value and stash it (/dev/sdc), then call connect_volume again (/dev/sda) to use it in the rescue instance, and then use the stashed value (/dev/sdc) that no longer exists
16:47:05 geguileo dansmith: yes, that's it
16:47:18 dansmith geguileo: okay
16:47:25 geguileo dansmith: if connect_volume is called, make sure that is used in all the instances that use the same cinder volume on that host
16:47:53 dansmith sounds like the solution is that we need to call connect_volume every time we're going to start an instance for any reason, and update the XML (rescue or otherwise) to use that value before we start
16:48:43 geguileo dansmith: that would be slower but safe
16:49:12 geguileo the optimised mechanism where we could tweak Nova to only do that when necessary could lead to unintended bugs
16:49:13 bauzas yeah, I was hoping some mechanism that would prevent the unnecessary roundtrip but if that's hard, then...
16:49:18 dansmith yeah I just don't know how we could do it safely otherwise, unless there's something that maintains an exclusive threadsafe set of host devices to make sure we never get them confused
16:50:02 geguileo dansmith: if the host doesn't reboot and connect_volume is not called again, then we could assume that the device is correct
16:50:13 bauzas geguileo: but we don't know thisq
16:50:27 bauzas geguileo: we only know about the service restart
16:50:40 geguileo bauzas: could nova query the uptime or something?
16:50:47 dansmith no :)
16:51:03 geguileo ok, then the whole connect_volume is probably the only way :'-(
16:51:08 dansmith let's not do anything like that for host or service uptime.. I think we just need to do the safe/slow approach
16:51:24 bauzas geguileo: we have other host-dependent resources like mdevs that we treat this way without requiring to know if it's a reboot
16:53:08 geguileo so I guess that's the solution then?
16:53:37 dansmith I think so.. not ideal for sure
16:53:43 bauzas in general, we follow some workflow which is "lookup my XML, find the specific resources attached, then query whatever needed to know whether those still exist, and if not, ask whatever else to recreate them"
16:54:08 bauzas (at service restart I mean)
16:54:35 geguileo with changes to cinder drivers we could have a proper check if it's valid, or have the connect-volume be faster (and idempotent when it can)
16:54:56 geguileo they would need to provide extra information to validate the device
16:55:10 dansmith let's do the slow/safe thing now, and if connect_volume() can be idempotent and faster in the future, then that's cool
16:55:20 geguileo dansmith: sounds like a plan
16:55:48 geguileo since that way the newer approach would probably not require additional nova changes
16:55:52 geguileo just os-brick and cinder
16:55:55 dansmith yep
16:56:59 geguileo ok, then I have nothing else to say on the topic
16:57:06 dansmith geguileo: unrelated, but since you're here.. what's cinder's plan for stable/train?
16:57:15 dansmith I assume no backports of this CVE back that far right?
16:57:23 dansmith and are you all thinking about EOLing at some point?
16:57:46 geguileo dansmith: definitely no backports that far
16:57:49 dansmith we have concerns about keeping branches open that look maintained but don't have backports of high-profile CVEs (two for us now)
16:58:08 geguileo I believe at some point it was discussed to stop supporting anything before Yoga
16:58:29 dansmith ack
16:58:36 bauzas anyway, we're on time
16:58:44 bauzas the nova meeting is ending in 1 min
16:59:07 dansmith thanks geguileo
16:59:16 geguileo thank you all
16:59:20 bauzas geguileo: dansmith: I guess we've arrived on a conclusion
16:59:25 bauzas thanks both of you
16:59:29 bauzas and thanks all
16:59:48 bauzas if nothing else,
16:59:53 bauzas bye
16:59:59 opendevmeet Log: https://meetings.opendev.org/meetings/nova/2023/nova.2023-05-30-16.00.log.html
16:59:59 opendevmeet Minutes (text): https://meetings.opendev.org/meetings/nova/2023/nova.2023-05-30-16.00.txt
16:59:59 opendevmeet Minutes: https://meetings.opendev.org/meetings/nova/2023/nova.2023-05-30-16.00.html
16:59:59 opendevmeet Meeting ended Tue May 30 16:59:59 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
16:59:59 bauzas #endmeeting
17:00:09 elodilles thanks o/
17:01:21 stephenfin Didn't bring it up in the meeting cos it doesn't belong there, but another round of +2s on the outstanding sqlalchemy-20 stuff would make my week and get us towards unblocking that in Bobcat. sean-k-mooney[m] has already spun through them
17:01:22 stephenfin All can be found here https://review.opendev.org/q/project:openstack/nova+topic:sqlalchemy-20+is:open
17:01:38 stephenfin Happy to answer any questions you may have on any/all of them
17:02:53 bauzas stephenfin: ack, no promises but I'll try to review them soon
17:03:01 bauzas stephenfin: bump me again next week if not
17:03:16 stephenfin will do, thanks bauzas
17:06:36 opendevreview Dan Smith proposed openstack/nova master: Add compute_id columns to instances, migrations https://review.opendev.org/c/openstack/nova/+/879499
17:06:36 opendevreview Dan Smith proposed openstack/nova master: Populate ComputeNode.service_id https://review.opendev.org/c/openstack/nova/+/879904
17:06:37 opendevreview Dan Smith proposed openstack/nova master: Add compute_id to Instance object https://review.opendev.org/c/openstack/nova/+/879500
17:06:37 opendevreview Dan Smith proposed openstack/nova master: Add dest_compute_id to Migration object https://review.opendev.org/c/openstack/nova/+/879682
17:06:38 opendevreview Dan Smith proposed openstack/nova master: Add online migration for Instance.compute_id https://review.opendev.org/c/openstack/nova/+/884752
17:06:38 opendevreview Dan Smith proposed openstack/nova master: Online migrate missing Instance.compute_id fields https://review.opendev.org/c/openstack/nova/+/879905
17:24:36 dansmith So, the last GPF kernel crash we had was a week ago
17:25:04 dansmith it was on the new cirros but it doesn't look exactly like what we saw before
17:25:27 dansmith but it does look related to acpi hotplug, which I assume is related to the volume attach or detach, and _is_ in one of those volume-having tests
17:26:33 dansmith so at this point I'd say the new cirros didn't fix it, but we'll have to wait for a few weeks of data to see if it's better or different
17:44:16 frickler dansmith: do you have a link to that crash handy? also we're right now publishing 0.6.2 with an updated kernel, that might help https://github.com/cirros-dev/cirros/actions/runs/5123918187
17:45:45 dansmith frickler: here's the 0.6.1 one: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ad6/826755/16/check/nova-lvm/ad65569/job-output.txt
17:45:51 dansmith search for "general protection"
17:48:37 opendevreview Sylvain Bauza proposed openstack/nova stable/xena: Fix get_segments_id with subnets without segment_id https://review.opendev.org/c/openstack/nova/+/883725
17:50:17 opendevreview Sylvain Bauza proposed openstack/nova stable/xena: Fix get_segments_id with subnets without segment_id https://review.opendev.org/c/openstack/nova/+/883725
17:53:22 opendevreview Sylvain Bauza proposed openstack/nova stable/yoga: Fix get_segments_id with subnets without segment_id https://review.opendev.org/c/openstack/nova/+/883724
17:55:05 opendevreview Sylvain Bauza proposed openstack/nova stable/xena: Fix get_segments_id with subnets without segment_id https://review.opendev.org/c/openstack/nova/+/883725
18:02:54 opendevreview Sylvain Bauza proposed openstack/nova stable/wallaby: Fix get_segments_id with subnets without segment_id https://review.opendev.org/c/openstack/nova/+/883726
20:57:01 opendevreview Merged openstack/nova master: db: Remove the legacy 'migration_version' table https://review.opendev.org/c/openstack/nova/+/872429
21:50:10 melwitt jjm9ASDFGHJJKOL;;;;;''''
21:51:33 melwitt geguileo: w2f
21:51:48 melwitt argh sorry

Earlier   Later