| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-02-22 | |||
| 14:59:59 | lyaaaaarwood | mdbooth: with migrate you provide the host (hostname@backend) you want to migrate to. With retype you provide a type you'd like to migrate to that can optionally contain a host (hostname@backend) extra spec that moves the volume | |
| 15:00:02 | mriedem | kashyap: i looked in the libvirtd log and didn't really see anything interesting | |
| 15:00:20 | mriedem | kashyap: besides stuff like it created the monitor for that instance and events weren't getting handled, something like that | |
| 15:01:23 | openstackgerrit | Adam Spiers proposed openstack/nova master: Add new "supports_amd_sev" capability to libvirt driver https://review.openstack.org/638680 | |
| 15:01:36 | kashyap | Nod; /me still downloading the log | |
| 15:09:12 | frickler | mriedem: anything I can do to help with https://bugs.launchpad.net/nova/+bug/1815082 ? | |
| 15:09:13 | openstack | Launchpad bug 1815082 in OpenStack Compute (nova) ""DBNonExistentTable: (sqlite3.OperationalError) no such table: services" when starting nova-metadata under uwsgi" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 15:09:30 | frickler | I'm also wondering whether this may be a duplicate actually https://bugs.launchpad.net/devstack/+bug/1814016 | |
| 15:09:31 | openstack | Launchpad bug 1814016 in devstack "Subnode required CELLSV2_SETUP=superconductor to be explicitly set" [Undecided,New] | |
| 15:15:46 | mriedem | frickler: ah i forgot about that | |
| 15:17:35 | frickler | mriedem: o.k., I'm assuming I did my part of the job, then. ;) | |
| 15:18:19 | mriedem | well n-api-meta is hitting conductor http://logs.openstack.org/77/635577/2/check/tempest-full/152e5bc/controller/logs/screen-n-api-meta.txt.gz#_Feb_08_16_53_34_572980 | |
| 15:18:32 | mriedem | i think | |
| 15:18:41 | mriedem | but it seems that the conductor it's hitting isn't up yet | |
| 15:20:15 | mriedem | http://logs.openstack.org/77/635577/2/check/tempest-full/152e5bc/controller/logs/devstacklog.txt.gz#_2019-02-08_16_52_32_215 | |
| 15:20:15 | mriedem | we start n-api-meta here | |
| 15:20:19 | frickler | http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/nova#n1096 | |
| 15:20:37 | frickler | that's in start_nova_rest, way before _conductor, yes | |
| 15:21:04 | mriedem | http://logs.openstack.org/77/635577/2/check/tempest-full/152e5bc/controller/logs/devstacklog.txt.gz#_2019-02-08_16_52_36_721 | |
| 15:21:04 | mriedem | yeah n-super-cond is started here: | |
| 15:21:44 | frickler | probably n-api-meta should idle a bit and retry | |
| 15:22:07 | frickler | even if we fix the order in devstack, we can't assume deployers will fine tune this everywhere | |
| 15:22:37 | mriedem | start_nova_rest is kind of weird - it starts nova-network as well which has always needed conductor to start first | |
| 15:22:39 | kashyap | mriedem: Oops: I think it is due to two 'pty' consoles getting reated...see lines 87-97 (from the n-cpu log): http://paste.openstack.org/show/745736/ | |
| 15:23:30 | mriedem | kashyap: would that be due to your recent version bump change? | |
| 15:23:34 | mriedem | which messed with the console code? | |
| 15:23:46 | kashyap | Yes, very much likely :-( | |
| 15:23:55 | kashyap | I fixed the s390x case, and I didn't see this at all in normal runs | |
| 15:23:56 | kashyap | Sigh | |
| 15:24:03 | kashyap | This seems to be 'isa-serial'-related | |
| 15:24:15 | mriedem | something something risk of doing this at the end of a release... | |
| 15:28:32 | kashyap | Yeah, I know what you mean. I made at least 3 folks to review, including the last person who reworked that code | |
| 15:28:55 | kashyap | I'll work towards fixing it as quickly as I can. | |
| 15:31:49 | mriedem | frickler: i think we're hitting the wrong conductor maybe | |
| 15:32:10 | mriedem | frickler: conductor is effectively started at 16:52:40 | |
| 15:32:14 | mriedem | n-super-cond that is | |
| 15:32:25 | mriedem | n-api-meta times out at 16:53:34 | |
| 15:32:38 | mriedem | note there is a 60 second rpc_response_timeout on that db query | |
| 15:33:46 | mriedem | problem is i'm not exactly sure which conductor n-api-meta is trying to hit | |
| 15:35:53 | mriedem | need to push a change to dump the config on startup before trying to hit the db | |
| 15:36:43 | kashyap | mriedem: Okay, one thing eliminated: that XML bit is a red-herring; it's not the cause. | |
| 15:36:51 | kashyap | (And the patch for consoles is still correct.) | |
| 15:37:36 | kashyap | (The seemingly two consoles is due to a silly backward compantbiility thing in libvirt) | |
| 15:40:01 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Set the conductor indirection API when running nova-metadata under uwsgi https://review.openstack.org/635577 | |
| 15:40:02 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Dump config options on wsgi startup earlier https://review.openstack.org/638691 | |
| 15:43:19 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: tox: Don't write byte code (maybe) https://review.openstack.org/636919 | |
| 15:47:49 | kashyap | mriedem: Do you know if these Ubuntu machines are using AppArmour? | |
| 15:48:38 | sean-k-mooney | kashyap: its installed by default so proably | |
| 15:49:15 | mriedem | kashyap: that's a question for infra | |
| 15:51:03 | kashyap | Okay | |
| 15:51:30 | kashyap | Talking to a libvirt dev, the core problem is QEMU failed to start, and consequently failed to create the socket. | |
| 15:51:36 | kashyap | Now trying to determine why | |
| 15:57:56 | kashyap | sean-k-mooney: Thanks | |
| 16:10:51 | giblet | mriedem: regarding your comments in https://review.openstack.org/#/c/569459/67/nova/network/neutronv2/api.py | |
| 16:11:04 | giblet | mriedem: i did not consider live-migration or other server move operation there | |
| 16:11:39 | giblet | mriedem: I think _update_port_binding_for_instance() needs to change when we enable server move operations | |
| 16:12:04 | giblet | mriedem: the RP uuid fulfilling a port's allocation will definitly change when we move the server to another host | |
| 16:18:56 | kashyap | mriedem: BTW, so this kind of completely silent failure is typically caused due to SELinux/AppArmour denials | |
| 16:19:11 | kashyap | But that's rather difficult to debug in OpenStack envs... | |
| 16:19:32 | kashyap | I'd be interested if this is reproducible at all -- should get to know it is by Monday | |
| 16:21:14 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove misleading comment from _move_operation_alloc_request() https://review.openstack.org/638700 | |
| 16:21:14 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Log why rescheduling is disabled https://review.openstack.org/638699 | |
| 16:21:15 | mriedem | leakypipes: dansmith: some fun from the past ^ | |
| 16:26:42 | sean-k-mooney | kashyap: wouldnt that show up in the audit.log | |
| 16:27:31 | kashyap | sean-k-mooney: No idea what AppArmour does; let me see, though | |
| 16:27:37 | kashyap | I know how to troubleshoot SELinuz | |
| 16:28:32 | sean-k-mooney | both used to show up in demesg then they moved to journalctl but were also loged to the audit.log under /var/log i think | |
| 16:31:01 | sean-k-mooney | this is also a kernel bug in ubuntu 18.04 that causes nested virt to not work but if that iss the issue the core dump in dmesg telegraphfs the issue rather quirckly. | |
| 16:32:09 | kashyap | sean-k-mooney: Where is the audit.log here: http://logs.openstack.org/48/631948/9/check/tempest-full-py3/e2ae3fb/controller/logs/ | |
| 16:32:10 | sean-k-mooney | kashyap: this is the ubuntu issue https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1773184 | |
| 16:32:11 | openstack | Launchpad bug 1773184 in linux (Ubuntu Bionic) "Starting a KVM guest in a guest (nested VM) crash the kernel" [High,Triaged] | |
| 16:32:31 | dansmith | bland? | |
| 16:32:35 | dansmith | blonde? | |
| 16:32:59 | kashyap | :D | |
| 16:33:13 | sean-k-mooney | i dont think its copied | |
| 16:33:13 | kashyap | Got dansmithed ... s/blnd/blind :D | |
| 16:33:33 | kashyap | Let me check with Clark | |
| 16:34:17 | giblet | mriedem: replied to your question in https://review.openstack.org/#/c/569459 | |
| 16:35:04 | kashyap | sean-k-mooney: BTW, upstream KVM list has a huge nested virt patch series; so no idea if that fixes | |
| 16:35:10 | kashyap | sean-k-mooney: I've seen that stack trace, though | |
| 16:35:13 | sean-k-mooney | kashyap: i have it https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1773184 on one of my servers byt the way so i know that iss an issue with the ubunut cloud images. that said i think its realteded to my bios/microcode | |
| 16:35:16 | openstack | Launchpad bug 1773184 in linux (Ubuntu Bionic) "Starting a KVM guest in a guest (nested VM) crash the kernel" [High,Triaged] | |
| 16:36:06 | sean-k-mooney | kashyap: well for me swaping out the vanial 4.20 kernel in the l1 guest allowed the l2 guest to boot | |
| 16:36:39 | sean-k-mooney | if i use centos on the same host for the l1 guest spanwnign the l2 guest crashes the l1 guest and it reboot | |
| 16:37:28 | kashyap | Okay, my cardinal rule when setting up nested virt for dev envs: ensure L0, L1 (and even L2) kernels to be as similiar (and as newer as they can) | |
| 16:37:50 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Use Selection object to fill request group mapping https://review.openstack.org/638711 | |
| 16:37:54 | sean-k-mooney | i was useing the same kernel for l0 and l1 originally | |
| 16:37:59 | kashyap | sean-k-mooney: The test combination explosion is one of the difficult things | |
| 16:38:02 | sean-k-mooney | with a cirros image for l2 | |
| 16:38:10 | sean-k-mooney | ya | |
| 16:38:27 | dansmith | mriedem: got it | |
| 16:38:55 | sean-k-mooney | i honestly have never had issue with nested vert in the past | |
| 16:40:01 | kashyap | sean-k-mooney: The "past" is a fuzzy word. Precisely for these reasons I used to track all the kernel bugs I filed for nVMX: https://kashyapc.fedorapeople.org/kernel-kvm-bugs.txt | |
| 16:40:18 | kashyap | (They're all fixed, though. And it hasn't been updated in ages) | |
| 16:45:08 | mriedem | the more i dig into https://bugs.launchpad.net/nova/+bug/1790204 the more it's a spec at this point | |
| 16:45:09 | openstack | Launchpad bug 1790204 in OpenStack Compute (nova) "Allocations are "doubled up" on same host resize even though there is only 1 server on the host" [High,Triaged] | |
| 16:47:14 | mriedem | i imagine this is all dead code since queens or at least rocky https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L140 | |
| 16:47:43 | mriedem | because we should never have to double up allocations on the same consumer (instance) during scheduling since we moved the allocations on the source node to the migration record, | |
| 16:47:43 | sean-k-mooney | kashyap: let me rephsase this is the first time i have ever personally it a nested vert bug in the 5-6 years i have used nested virt by defualt | |
| 16:47:47 | mriedem | except maybe evacuate... | |