Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-02-22
15:21:44 frickler probably n-api-meta should idle a bit and retry
15:22:07 frickler even if we fix the order in devstack, we can't assume deployers will fine tune this everywhere
15:22:37 mriedem start_nova_rest is kind of weird - it starts nova-network as well which has always needed conductor to start first
15:22:39 kashyap mriedem: Oops: I think it is due to two 'pty' consoles getting reated...see lines 87-97 (from the n-cpu log): http://paste.openstack.org/show/745736/
15:23:30 mriedem kashyap: would that be due to your recent version bump change?
15:23:34 mriedem which messed with the console code?
15:23:46 kashyap Yes, very much likely :-(
15:23:55 kashyap I fixed the s390x case, and I didn't see this at all in normal runs
15:23:56 kashyap Sigh
15:24:03 kashyap This seems to be 'isa-serial'-related
15:24:15 mriedem something something risk of doing this at the end of a release...
15:28:32 kashyap Yeah, I know what you mean. I made at least 3 folks to review, including the last person who reworked that code
15:28:55 kashyap I'll work towards fixing it as quickly as I can.
15:31:49 mriedem frickler: i think we're hitting the wrong conductor maybe
15:32:10 mriedem frickler: conductor is effectively started at 16:52:40
15:32:14 mriedem n-super-cond that is
15:32:25 mriedem n-api-meta times out at 16:53:34
15:32:38 mriedem note there is a 60 second rpc_response_timeout on that db query
15:33:46 mriedem problem is i'm not exactly sure which conductor n-api-meta is trying to hit
15:35:53 mriedem need to push a change to dump the config on startup before trying to hit the db
15:36:43 kashyap mriedem: Okay, one thing eliminated: that XML bit is a red-herring; it's not the cause.
15:36:51 kashyap (And the patch for consoles is still correct.)
15:37:36 kashyap (The seemingly two consoles is due to a silly backward compantbiility thing in libvirt)
15:40:01 openstackgerrit Matt Riedemann proposed openstack/nova master: Set the conductor indirection API when running nova-metadata under uwsgi https://review.openstack.org/635577
15:40:02 openstackgerrit Matt Riedemann proposed openstack/nova master: Dump config options on wsgi startup earlier https://review.openstack.org/638691
15:43:19 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: tox: Don't write byte code (maybe) https://review.openstack.org/636919
15:47:49 kashyap mriedem: Do you know if these Ubuntu machines are using AppArmour?
15:48:38 sean-k-mooney kashyap: its installed by default so proably
15:49:15 mriedem kashyap: that's a question for infra
15:51:03 kashyap Okay
15:51:30 kashyap Talking to a libvirt dev, the core problem is QEMU failed to start, and consequently failed to create the socket.
15:51:36 kashyap Now trying to determine why
15:57:56 kashyap sean-k-mooney: Thanks
16:10:51 giblet mriedem: regarding your comments in https://review.openstack.org/#/c/569459/67/nova/network/neutronv2/api.py
16:11:04 giblet mriedem: i did not consider live-migration or other server move operation there
16:11:39 giblet mriedem: I think _update_port_binding_for_instance() needs to change when we enable server move operations
16:12:04 giblet mriedem: the RP uuid fulfilling a port's allocation will definitly change when we move the server to another host
16:18:56 kashyap mriedem: BTW, so this kind of completely silent failure is typically caused due to SELinux/AppArmour denials
16:19:11 kashyap But that's rather difficult to debug in OpenStack envs...
16:19:32 kashyap I'd be interested if this is reproducible at all -- should get to know it is by Monday
16:21:14 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove misleading comment from _move_operation_alloc_request() https://review.openstack.org/638700
16:21:14 openstackgerrit Matt Riedemann proposed openstack/nova master: Log why rescheduling is disabled https://review.openstack.org/638699
16:21:15 mriedem leakypipes: dansmith: some fun from the past ^
16:26:42 sean-k-mooney kashyap: wouldnt that show up in the audit.log
16:27:31 kashyap sean-k-mooney: No idea what AppArmour does; let me see, though
16:27:37 kashyap I know how to troubleshoot SELinuz
16:28:32 sean-k-mooney both used to show up in demesg then they moved to journalctl but were also loged to the audit.log under /var/log i think
16:31:01 sean-k-mooney this is also a kernel bug in ubuntu 18.04 that causes nested virt to not work but if that iss the issue the core dump in dmesg telegraphfs the issue rather quirckly.
16:32:09 kashyap sean-k-mooney: Where is the audit.log here: http://logs.openstack.org/48/631948/9/check/tempest-full-py3/e2ae3fb/controller/logs/
16:32:10 sean-k-mooney kashyap: this is the ubuntu issue https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1773184
16:32:11 openstack Launchpad bug 1773184 in linux (Ubuntu Bionic) "Starting a KVM guest in a guest (nested VM) crash the kernel" [High,Triaged]
16:32:31 dansmith bland?
16:32:35 dansmith blonde?
16:32:59 kashyap :D
16:33:13 sean-k-mooney i dont think its copied
16:33:13 kashyap Got dansmithed ... s/blnd/blind :D
16:33:33 kashyap Let me check with Clark
16:34:17 giblet mriedem: replied to your question in https://review.openstack.org/#/c/569459
16:35:04 kashyap sean-k-mooney: BTW, upstream KVM list has a huge nested virt patch series; so no idea if that fixes
16:35:10 kashyap sean-k-mooney: I've seen that stack trace, though
16:35:13 sean-k-mooney kashyap: i have it https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1773184 on one of my servers byt the way so i know that iss an issue with the ubunut cloud images. that said i think its realteded to my bios/microcode
16:35:16 openstack Launchpad bug 1773184 in linux (Ubuntu Bionic) "Starting a KVM guest in a guest (nested VM) crash the kernel" [High,Triaged]
16:36:06 sean-k-mooney kashyap: well for me swaping out the vanial 4.20 kernel in the l1 guest allowed the l2 guest to boot
16:36:39 sean-k-mooney if i use centos on the same host for the l1 guest spanwnign the l2 guest crashes the l1 guest and it reboot
16:37:28 kashyap Okay, my cardinal rule when setting up nested virt for dev envs: ensure L0, L1 (and even L2) kernels to be as similiar (and as newer as they can)
16:37:50 openstackgerrit Balazs Gibizer proposed openstack/nova master: Use Selection object to fill request group mapping https://review.openstack.org/638711
16:37:54 sean-k-mooney i was useing the same kernel for l0 and l1 originally
16:37:59 kashyap sean-k-mooney: The test combination explosion is one of the difficult things
16:38:02 sean-k-mooney with a cirros image for l2
16:38:10 sean-k-mooney ya
16:38:27 dansmith mriedem: got it
16:38:55 sean-k-mooney i honestly have never had issue with nested vert in the past
16:40:01 kashyap sean-k-mooney: The "past" is a fuzzy word. Precisely for these reasons I used to track all the kernel bugs I filed for nVMX: https://kashyapc.fedorapeople.org/kernel-kvm-bugs.txt
16:40:18 kashyap (They're all fixed, though. And it hasn't been updated in ages)
16:45:08 mriedem the more i dig into https://bugs.launchpad.net/nova/+bug/1790204 the more it's a spec at this point
16:45:09 openstack Launchpad bug 1790204 in OpenStack Compute (nova) "Allocations are "doubled up" on same host resize even though there is only 1 server on the host" [High,Triaged]
16:47:14 mriedem i imagine this is all dead code since queens or at least rocky https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L140
16:47:43 mriedem because we should never have to double up allocations on the same consumer (instance) during scheduling since we moved the allocations on the source node to the migration record,
16:47:43 sean-k-mooney kashyap: let me rephsase this is the first time i have ever personally it a nested vert bug in the 5-6 years i have used nested virt by defualt
16:47:47 mriedem except maybe evacuate...
16:48:00 kashyap sean-k-mooney: Ah, okay. :-)
16:49:00 melwitt o/
16:59:56 cdent fried_rice: my turn to ask you for a reminder:
17:00:20 cdent resource provider A represents a compute node, currently idle
17:00:58 cdent third part system makes allocations against that compute node (consuming capacity in placement but not in _reality_)
17:01:31 cdent the compute node will or will not correct that?
17:01:57 cdent I feel like we've gone back and forth on that a bit and I forget where we are now
17:02:11 cdent I'd just test it, but I currently have no cloud
17:03:30 fried_rice cdent: If the allocation is made against a real instance UUID, I think nova will "heal" it at some point. Otherwise, we don't muck with allocations that aren't related to instances afaik.
17:03:58 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Dedupe compute install guides https://review.openstack.org/638716
17:03:58 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Dedupe controller install guides https://review.openstack.org/638715
17:04:17 fried_rice cdent: mriedem might have to weigh in on the circumstances under which healing occurs at all. It may be just at the behest of a nova-manage command.
17:04:41 cdent i've managed to make myself curious enough that I'm going to go spin up the devstack
17:05:04 fried_rice cdent: Now, oob inventory/trait/agg changes against providers owned by the compute node will definitely be "healed" next time upt runs.
17:05:32 fried_rice which actually may not be as soon as we would like - was having this conversation with aspiers the other day.
17:06:36 cdent yeah, to inventory/trait stuff
17:07:01 cdent what's the issue on the "soon"?
17:09:25 mriedem there is no auto-healing of allocations by nova-compute
17:09:30 kashyap mriedem: I think I found the sucker in my case; spent the last hour-ish duking aroud the system. This looks suspicious:
17:09:34 kashyap ---

Earlier   Later