Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-11-05
22:05:44 sean-k-mooney im just not passing the correct flags when building it to have it auto install the apparmor files
22:06:06 sean-k-mooney i should prably just ask danpb how to do it correctly
22:11:18 melwitt mriedem: do you remember any func test change from the past made to avoid "{"forbidden": {"code": 403, "message": "Maximum number of ports exceeded"}}"
22:11:55 mriedem are you backporting something to queens or newton or something and hitting that?
22:11:57 melwitt I'm backporting stuff and hitting this because I'm presumably missing some underlying func test improvement
22:11:58 melwitt yes
22:12:24 melwitt I thought if anyone might remember something related to that, it would be you
22:12:39 mriedem not specifically, but i'd look for port quota stuff on the NeutronFixture
22:12:54 melwitt thanks, any direction helps a lot :)
22:13:11 mriedem https://review.opendev.org/#/q/I1dbccc2be6ba79bf267edac9208c80e187e6256a ?
22:13:36 mriedem https://review.opendev.org/#/c/587412/4/nova/tests/fixtures.py@1347 is the thing i'd think you want
22:14:23 melwitt looks promising
22:25:08 sean-k-mooney efried: ok i think i have fixed the issue locally ill work on a patch. but basically i need to tweak the /etc/libvirt/libvirtd.conf sligtly and nova will be happy
22:25:31 sean-k-mooney efried: http://paste.openstack.org/show/785827/
22:27:02 openstackgerrit Merged openstack/nova master: Follow up to I3e28c0163dc14dacf847c5a69730ba2e29650370 https://review.opendev.org/692856
22:27:04 efried sean-k-mooney: okay. I tried doing that conf change and now it's doing this
22:27:04 efried Connection to libvirt failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory: libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
22:28:12 sean-k-mooney where is that error? nova? virsh?
22:28:47 efried nova compute startup (journalctl)
22:29:15 sean-k-mooney ok what do you get if you do "sudo systemctl status libvirt-bin"
22:30:34 openstackgerrit Matt Riedemann proposed openstack/nova master: Improve metadata server performance with large security groups https://review.opendev.org/656084
22:30:37 mriedem this is a simple performance improvement fix, been sitting a long time waiting for the originaly author to touch it up ^
22:30:46 mriedem and when i went to do it, i realized there was a much simpler fix
22:31:30 efried d'oh, sorry, already started unstacking.
22:31:56 sean-k-mooney no worries
22:32:07 efried sean-k-mooney: I'm adding those extra libvirtd.conf lines (from within the plugin) and will restack.
22:32:32 sean-k-mooney by the way i have code in the pluging to fix the app armor stuff
22:32:34 sean-k-mooney sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtd || /bin/true
22:32:39 sean-k-mooney im guessing that is failing
22:32:48 sean-k-mooney because i thinkg the package name chagned
22:32:55 sean-k-mooney so im fining that too
22:33:00 efried are we not set -e here?
22:33:27 efried oh, || true would ignore :(
22:33:34 sean-k-mooney yes
22:34:06 sean-k-mooney the /bin/ture was to ignore it on subsequent runs but its really just a hack
22:35:03 sean-k-mooney install_build_deps for libvirt-bin used to pull in the apparmor utils but i apparently does not anymore so ill install them direcly
22:35:16 sean-k-mooney im guessing you dont have apparmor_parser right
22:36:34 sean-k-mooney i should not really depend on the transitive depencies like that
22:37:58 efried sean-k-mooney: I have an apparmor_parser in /sbin
22:38:44 sean-k-mooney hum ok it should have remvoed the proifle then but ill document the other way in anycase
22:39:41 efried the file it's referring to doesn't exist fwiw
22:41:07 melwitt mriedem: you were right, that solved the 403. now it's on to 409 "Multiple possible networks found, use a Network ID to be more specific". I'm sifting through NeutronFixture changes
22:41:54 efried sean-k-mooney: with those extra lines, now stack fails when libvirt-bin won't start.
22:41:55 sean-k-mooney efried: there is still a profile in /etc/apparmor.d/libvirt/ in my case
22:42:32 efried that line is skipping
22:42:32 efried +/opt/stack/devstack-plugin-libvirt-qemu/devstack/libs/libvirt:install_libvirt_qemu_src:210 sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtd
22:42:32 efried File /etc/apparmor.d/usr.sbin.libvirtd not found, skipping...
22:43:05 efried the libvirt-bin service is failing thusly:
22:43:05 efried Nov 05 16:40:14 nucle systemd[1]: libvirtd.service: Service hold-off time over, scheduling restart.
22:43:05 efried Nov 05 16:40:14 nucle systemd[1]: libvirtd.service: Scheduled restart job, restart counter is at 5.
22:43:05 efried Nov 05 16:40:14 nucle systemd[1]: Stopped Virtualization daemon.
22:43:05 efried Nov 05 16:40:14 nucle systemd[1]: libvirtd.service: Start request repeated too quickly.
22:43:05 efried Nov 05 16:40:14 nucle systemd[1]: libvirtd.service: Failed with result 'exit-code'.
22:43:06 efried Nov 05 16:40:14 nucle systemd[1]: Failed to start Virtualization daemon.
22:43:08 sean-k-mooney ya it has obviously moved
22:43:45 efried I see these
22:43:45 efried /etc/apparmor.d/cache/usr.sbin.libvirtd
22:43:45 efried /opt/stack/libvirt/src/security/apparmor/usr.sbin.libvirtd
22:43:46 sean-k-mooney do you have anything in the journal
22:44:18 efried Assuming I would find it with journalctl -u libvirt-bin, no
22:44:35 sean-k-mooney it would be libvirtd-bin
22:44:43 sean-k-mooney but i dont think that is where its logging
22:44:48 efried either way
22:44:49 efried no
22:45:03 sean-k-mooney ok give me a sec to push this up and test it
22:45:14 efried crap, I f'ed up the libvirtd.conf hack, sec.
22:47:50 melwitt mriedem: ok, looks like I just needed more of that change and now things are working. gonna try backporting the whole thing
22:48:57 mriedem melwitt: which target branch? is this upstream?
22:49:06 mriedem because i don't think we want to backport that whole thing...
22:50:09 melwitt mriedem: this is downstream queens. really? ok
22:50:19 mriedem well, it's big
22:50:26 mriedem note you already have an abandoned backport of that upstream on stable/queens
22:50:38 mriedem https://review.opendev.org/#/c/599764/
22:51:05 melwitt cause it looks like I need the bits removing the validate_networks stub, add the shared network stuff, and add stuff from this change too https://review.opendev.org/#/c/585385/15/nova/tests/fixtures.py
22:51:18 melwitt lol welp
22:51:25 mriedem you can't just do fake_network.set_stub_network_methods again?
22:51:42 melwitt maybe I can. I have zero memory of doing that before
22:51:45 melwitt I'll try it
22:52:05 melwitt that would be way easier
22:54:09 sean-k-mooney efried: im testing https://review.opendev.org/#/c/693094/1 now
22:55:02 sean-k-mooney efried: assuming that correct the issue ill see about setting up a tempest gate jobs that uses it and then you can just reuse/extend it for your own stuff
23:05:31 openstackgerrit Takashi NATSUME proposed openstack/python-novaclient master: doc: Fix supported version descriptions https://review.opendev.org/693097
23:25:58 sean-k-mooney efried: looks like i should be adding "--with-apparmor_profiles" when compiling to have it install them correctly.
23:26:54 sean-k-mooney of cource the only mention of that flag in the libvirt repo is in the configure script ...
23:27:30 sean-k-mooney sometimes i forget how "fun" C is
23:34:26 efried sean-k-mooney: how do I check my version of qemu?
23:35:00 efried n-cpu failed to start with
23:35:00 efried Nova requires QEMU version 2.8.0 or greater.
23:35:00 efried but the plugin should have set up 4.1.0, no?
23:35:35 sean-k-mooney correct
23:35:41 sean-k-mooney http://paste.openstack.org/show/785829/
23:35:50 sean-k-mooney i have 4.1
23:39:31 openstackgerrit melanie witt proposed openstack/nova stable/stein: Add regression test for bug 1824435 https://review.opendev.org/692906
23:39:31 openstack bug 1824435 in OpenStack Compute (nova) stein "fill_virtual_interface_list migration fails on second attempt" [Medium,In progress] https://launchpad.net/bugs/1824435 - Assigned to melanie witt (melwitt)
23:39:31 openstackgerrit melanie witt proposed openstack/nova stable/stein: Remove redundant call to get/create default security group https://review.opendev.org/692907
23:45:43 sean-k-mooney efried: ill try running this all again on a clean ububut cloud image in the morning and ill provide you with the local.conf and anychagnes that are needed on the system if any in an update to the readme.
23:46:17 sean-k-mooney one way or another its in my best interest to get this documented and updated in the repo because i will need this again at somepoint in the futrue
23:53:16 efried sean-k-mooney: I gotta run and break some arms, but when I run the qemu commands I see 4.1, but obv nova thinks I've got something lesser
23:53:27 efried anyway, thanks for the help so far, I'll bug you again tomorrow.
23:54:32 sean-k-mooney no worries sorry it didnt "just work"

Earlier   Later