Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-11-06
15:24:18 sean-k-mooney mriedem: does http://paste.openstack.org/show/785852/ look familar
15:24:33 sean-k-mooney igrate.exceptions.VersionNotFoundError: \"Database schema file with version 72 doesn't exist.\"",
15:24:55 sean-k-mooney im not sure where the version 72 comes form in the code
15:26:01 sean-k-mooney oh its here https://github.com/openstack/nova/tree/master/nova/db/sqlalchemy/api_migrations/migrate_repo/versions
15:38:06 dansmith sean-k-mooney: yeah, those are where the indexes are
15:38:30 dansmith sean-k-mooney: usually that error happens when you have a pyc of the migration but not the py of it, because you have a dirty working directory or something
15:39:01 sean-k-mooney dansmith: in this case its becaue i deploy kolla with master after you added the placeholders
15:39:17 sean-k-mooney but the current master contaienr are now clamped to traing
15:39:23 sean-k-mooney *train
15:39:32 sean-k-mooney while they are preparing an rc
15:39:43 sean-k-mooney so the api sysnc is trying to do a downgrade
15:39:52 dansmith sure, but.. the only thing that causes it to know about migration 72 is the fact that that file exists on disk
15:39:55 sean-k-mooney which obviously wont work
15:40:03 dansmith downgrade?
15:40:27 sean-k-mooney they dont exist on stable train right
15:40:30 dansmith that shouldn't cause that error
15:40:49 dansmith unless you have a database with version 72 and are trying to run downgrade I guess
15:40:56 mriedem downgrade isn't around anymore
15:41:19 mriedem sdague ripped it out a long time ago
15:41:25 dansmith mriedem: it's just a pass and technically runs IIRC
15:41:25 sean-k-mooney ya
15:41:36 sean-k-mooney its fine
15:41:38 mriedem the method doesn't exist in the scripts anymore
15:41:42 sean-k-mooney its just my home openstack
15:41:43 dansmith but I dunno why anything would be trying to do it
15:42:02 dansmith mriedem: we know that
15:42:03 sean-k-mooney it was bad timing that i deploy master just after nova rc1 when that was merged
15:42:29 dansmith sean-k-mooney: so yeah must be that sqla-migrate sees the db version as 72 and is looking for the downgrade script I guess? but..that'd be your problem
15:42:38 sean-k-mooney but the current "master" kolla image are stable train as they have not done there rc1 yet
15:43:18 sean-k-mooney ya i can just build the container form master nova and continue or redeploy with stable train
15:43:18 mriedem heh so you time traveled
15:43:33 sean-k-mooney its not a big deal
15:44:50 openstackgerrit Matt Riedemann proposed openstack/nova stable/stein: Add --dry-run option to heal_allocations CLI https://review.opendev.org/693199
15:44:51 openstackgerrit Matt Riedemann proposed openstack/nova stable/stein: Add BFV wrinkle to TestNovaManagePlacementHealAllocations https://review.opendev.org/693201
15:44:51 openstackgerrit Matt Riedemann proposed openstack/nova stable/stein: Add --instance option to heal_allocations https://review.opendev.org/693200
15:46:28 sean-k-mooney well now i know where to look and how that works. im off to lunch brb
15:53:27 openstackgerrit Boris Bobrov proposed openstack/nova master: Also enable iommu for virtio controllers and video in libvirt https://review.opendev.org/684825
15:53:28 openstackgerrit Boris Bobrov proposed openstack/nova master: Create a controller for qga when SEV is used https://review.opendev.org/693072
16:04:06 openstackgerrit Boris Bobrov proposed openstack/nova master: Move rng device checks to the appropriate method https://review.opendev.org/693210
16:07:13 efried Nov 06 09:22:55 breakfast dnsmasq[27199]: exiting on receipt of SIGTERM
16:07:13 efried Nov 06 09:19:25 breakfast dnsmasq-dhcp[27199]: read /var/lib/libvirt/dnsmasq/default.hostsfile
16:07:13 efried The libvirt-bin service is inactive. There's no journal, but systemctl shows these as the last two lines:
16:07:13 efried '/var/run/libvirt/libvirt-sock': No such file or directory.
16:07:13 efried sean-k-mooney: back to n-cpu failing to start because of
16:07:14 efried I don't see anything in the stack logs that looks like it ought to have shut the service down.
16:08:22 efried On both stacks the systemctl status shows that ~3.5min gap between loading default.hostsfile and SIGTERM
16:13:43 efried sean-k-mooney: I was able to manually start libvirt-bin and it hasn't stopped itself, so clearly something else was hitting it...
16:14:50 efried ...but with the service running, n-cpu still fails to come up in the same way.
16:21:40 openstackgerrit Boris Bobrov proposed openstack/nova master: Create a controller for qga when SEV is used https://review.opendev.org/693072
16:21:40 openstackgerrit Boris Bobrov proposed openstack/nova master: Also enable iommu for virtio controllers and video in libvirt https://review.opendev.org/684825
16:22:26 mriedem amodi: melwitt: looks like we can keep the iso file without changes https://review.opendev.org/#/c/693065/7
16:22:41 mriedem it was already fixed in the manuals repo before i imported that doc https://github.com/openstack/openstack-manuals/commit/1206b9b727594ae04f0d2a4818f183e95253b464
16:22:43 openstackgerrit Merged openstack/nova master: Pass exception through TaskBase.rollback https://review.opendev.org/692689
16:25:55 openstackgerrit Matt Riedemann proposed openstack/nova master: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693065
16:27:00 mriedem this is why i'm not as free willy on accepting any old docs change since it's a pain in the butt to fix them if they are wrong
16:30:06 melwitt haha free willy
16:31:28 melwitt and yeah, understood. thanks for pointing out those previous fixes and updating it
16:34:12 dansmith mriedem: query
16:34:27 dansmith was just looking at your heal_instance_mappings thing
16:34:32 dansmith and was thinking.
16:35:57 dansmith I wonder if it wouldn't make sense for us to have another totally optional service with the sole purpose of looking for things that need healing.. instance mappings, maybe allocations, host mappings, maybe even archive and purge
16:36:31 dansmith we could make it a daemon, or something you could run from cron single-shot if you just wanted to run it at midnight or something
16:36:41 melwitt I think operators would love that
16:37:15 dansmith instead of a ton of cron entries for all those things, and not knowing you need new cron entries to run new things on a new release,
16:37:25 dansmith you just run one nova-audit in one place, no HA, just low and slow burn of audity things
16:37:40 melwitt yeah, would be nice
16:38:43 mriedem i'm not opposed
16:39:40 mriedem would that also run the discover host mappings thing that the scheduler can run today?
16:40:09 mriedem i remember glance has several things like that https://github.com/openstack/glance/blob/master/setup.cfg#L34
16:40:17 mriedem though i'm not sure if they are all daemons
16:40:22 dansmith mriedem: yeah, hence the host mappings in my list
16:41:41 mriedem i will admit the number of heal/discover/sync things is getting a bit overwhelming
16:41:49 dansmith right
16:42:00 dansmith impossible to track for the average operator I'm sure
16:42:03 mriedem and we have talked about a periodic archive/purge thing in the past
16:42:25 dansmith and we put the host mappings thing in scheduler mostly because the cardinality was small
16:42:38 dansmith (and then I subsequently solved their problem a different way)
16:42:59 mriedem and i think b/c api and conductor didn't have periodic tasks already but scheduler did
16:43:53 dansmith maybe, but the point was to run one or few of those since they'd step all over each other, and there'd be too many conductors and apis even if they did have periodics
16:43:54 mriedem maybe you want to kick that idea out into the mailing list tagged with [nova][ops] to get initial feedback? and/or just start a spec and kick that into the ML to get attention from operators.
16:44:23 mriedem we do run nproc/2 scheduler works by default now i think..
16:44:24 mriedem but yeah
16:44:28 dansmith yeah, I could start with a spec
16:44:38 dansmith I was kinda thinking of just starting with code
16:45:04 mriedem now that you've gotten a taste from the image cache stuff you're addicted!
16:45:55 dansmith JUST ONE MORE HIT MAN
16:48:44 efried Nov 06 10:45:03 breakfast nova-compute[15654]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python3.6/dist-packages/libvirt.py", line 4190, in getVersion
16:48:44 efried If I start n-cpu in that window, I run into the qemu version check. I added some debug and find it's coming from:
16:48:44 efried sean-k-mooney: The libvirt-bin service seems to not want to stay alive. When I start it manually, it runs for a couple minutes and then dies, but doesn't log anything.
16:48:45 efried Nov 06 10:45:03 breakfast nova-compute[15654]: ERROR nova.virt.libvirt.host libvirt.libvirtError: internal error: Cannot find suitable emulator for x86_64
16:48:45 efried Nov 06 10:45:03 breakfast nova-compute[15654]: ERROR nova.virt.libvirt.host if ret == -1: raise libvirtError ('virConnectGetVersion() failed', conn=self)
16:50:16 sean-k-mooney is qemu-system-x86_64
16:50:19 sean-k-mooney in your path
16:50:40 sean-k-mooney also do you have permission denied errors in dmesg
16:51:12 openstackgerrit Merged openstack/nova master: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693065
16:51:28 efried [73117.887250] audit: type=1400 audit(1573058703.103:556): apparmor="DENIED" operation="exec" profile="/usr/sbin/libvirtd" name="/usr/local/bin/qemu-system-microblazeel" pid=15694 comm="libvirtd" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
16:51:28 efried you mean like this
16:51:35 sean-k-mooney yes
16:51:46 sean-k-mooney so apparmor is blocking it
16:51:48 efried (yes, qemu-[system-]x86_64 is in /usr/local/bin, which is in my PATH, at least on the CLI)

Earlier   Later