| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-28 | |||
| 16:06:48 | sean-k-mooney | stephenfin: ya its in rc3 | |
| 16:06:50 | stephenfin | mriedem, sean-k-mooney: And there's the fix (tl;dr: this can has some kicking left in it) | |
| 16:07:32 | Tahvok | dansmith: all of them are actually replies as you can see | |
| 16:07:34 | stephenfin | sean-k-mooney: How'd you figure that out? | |
| 16:08:03 | dansmith | Tahvok: that doesn't really tell us anything | |
| 16:08:06 | sean-k-mooney | git fetch --tags && git checkout 18.0.0.0rc3 | |
| 16:08:17 | sean-k-mooney | then git log and search for change id | |
| 16:08:45 | Tahvok | dansmith: doesn't a reply message indicate it's a sync from compute nodes? | |
| 16:08:56 | stephenfin | sean-k-mooney: Ah, 'git branch -a --contains 2d532963fa2e013e16cc403f2674a4488c4170ab' | |
| 16:09:09 | stephenfin | The '-a' is important. I don't have stable/rocky locally yet | |
| 16:09:12 | sean-k-mooney | stephenfin: also if you look at https://github.com/openstack/nova/commit/2d532963fa2e013e16cc403f2674a4488c4170ab#diff-8fec546e4c39f78d233f8e21dadaa3ff it shows what branches have it | |
| 16:09:43 | dansmith | Tahvok: I don't think those lines are telling you that they're replies, they're telling you what queue will be used for the reply | |
| 16:11:01 | sean-k-mooney | stephenfin: mriedem in anycase https://review.openstack.org/#/c/597173/1 will need to get applied to stable/rocky | |
| 16:11:25 | mriedem | yes i know | |
| 16:11:27 | mriedem | i left a comment in there | |
| 16:11:45 | mriedem | we can't do an rc4 so i guess this is just going to be a known broken issue for anyone not using oslo.db 4.40 which is at least in upper-constraints for stable/rocky | |
| 16:12:03 | mriedem | and i'd think/hope most deployments should be using what's in upper-constraints for dependent libraries since those are the versions we test against | |
| 16:13:52 | melwitt | . | |
| 16:14:20 | sean-k-mooney | mriedem: yeah thats unfortunate but at least as you said upper constratins allows 4.4 | |
| 16:14:42 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Don't use '_TransactionContextManager._async' https://review.openstack.org/597173 | |
| 16:14:44 | sean-k-mooney | *4.40 | |
| 16:16:04 | efried | stephenfin: Getting late for you, you want me to propose the fup? | |
| 16:17:00 | melwitt | dansmith: cool, will look | |
| 16:18:05 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Revert "Don't use '_TransactionContextManager._async'" https://review.openstack.org/597174 | |
| 16:18:07 | stephenfin | efried: Cheers but it was pretty easy :) | |
| 16:18:48 | efried | stephenfin: Cool. Can you put Related-Bug in the commit? | |
| 16:19:11 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Revert "Don't use '_TransactionContextManager._async'" https://review.openstack.org/597174 | |
| 16:19:16 | stephenfin | efried: done | |
| 16:19:18 | efried | Thanks. +2 | |
| 16:32:05 | openstackgerrit | Merged openstack/nova master: Make scheduler.utils.setup_instance_group query all cells https://review.openstack.org/540258 | |
| 16:39:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Further simplification of test_numa_servers https://review.openstack.org/596832 | |
| 16:39:40 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Validate huge pages https://review.openstack.org/399653 | |
| 16:46:24 | openstackgerrit | Merged openstack/nova master: Record cell success/failure/timeout in CrossCellLister https://review.openstack.org/594265 | |
| 17:09:38 | sean-k-mooney | oh by the way i finally got around to rebasing https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bug/1759420 if people have time to review | |
| 17:17:06 | Tahvok | Is it safe to reduce the number of conductor workers | |
| 17:17:13 | Tahvok | ? | |
| 17:17:37 | Tahvok | I'm only afraid if I would lose some data.. | |
| 17:21:54 | sean-k-mooney | Tahvok: i belive if you want to reduce the number of workers you will need to restart the nova-conductor. | |
| 17:22:07 | Tahvok | sean-k-mooney: yeah, that's not the problem.. | |
| 17:22:27 | Tahvok | Just done it, I wanted a more organized log: http://paste.openstack.org/show/Iz2EukH6oxZnmdK3OKDX/ | |
| 17:22:35 | sean-k-mooney | if the condoctor has dequeue messages but not acked them then rabbit will keep them in the queues | |
| 17:22:56 | sean-k-mooney | other then that you should not loose any data | |
| 17:35:47 | dansmith | gdi, I can't get a running devstack to save my life | |
| 17:38:29 | mriedem | saml? | |
| 17:39:21 | mriedem | dansmith: i've heard other people reporting devstack failing b/c not being able to install pysaml | |
| 17:40:28 | dansmith | no, I think the problem is n-api isn't getting setup in the wsgi container, | |
| 17:40:39 | dansmith | so the n-api log is all kinds of missing python stuff | |
| 17:40:45 | sean-k-mooney | mriedem: ya had issues on centos 7 pysaml2 would not install from pip | |
| 17:40:49 | dansmith | hmm, maybe this is a py3 thing? | |
| 17:42:05 | sean-k-mooney | dansmith: if you are hitting the pysaml2 issue i have locally capped it to 4.5.0 | |
| 17:42:30 | sean-k-mooney | 4.6.0 was complaining about direcories not existing | |
| 17:44:31 | mriedem | sean-k-mooney: in -dev they said you needed newer pip/setuptools | |
| 17:44:56 | sean-k-mooney | mriedem: devstack on centos7 keeps downgrading my pip version | |
| 17:45:19 | sean-k-mooney | it seams to be pinning me to pip 9.0.3 | |
| 17:48:33 | sean-k-mooney | mriedem: i think devstack is prefering system python which is rather old on centos 7 | |
| 17:49:52 | sean-k-mooney | that said i tought i pulled down pip directly form pypi so i dont know why im stuck on version 9 | |
| 17:51:42 | sean-k-mooney | oh there is a tools/cap-pip.txt with pip!=8,<10 | |
| 17:52:07 | sean-k-mooney | mriedem: did they say waht version you need? | |
| 17:54:05 | dansmith | sean-k-mooney: okay I deleted enough stuff to get to the saml thing | |
| 17:54:08 | dansmith | so what's the workaround? | |
| 17:54:39 | sean-k-mooney | dansmith: mine is to modify the upper-constatints file to cap it to 4.5.0 | |
| 17:55:07 | dansmith | um modify where? | |
| 17:55:20 | sean-k-mooney | the other workaround would appear to be use a newer pip/setuptools but devstack caps pip to pip!=8,<10 in tools/cap-pip.txt | |
| 17:55:20 | dansmith | opt/requirements? | |
| 17:55:22 | dansmith | will that stick? | |
| 17:55:35 | sean-k-mooney | dansmith: yes /opt/requiremets | |
| 17:55:53 | sean-k-mooney | it will stick if you dont have RECLONE=Ture set | |
| 17:57:15 | dansmith | ack, thanks, trying | |
| 17:59:37 | dansmith | okay I think I'm past that point now | |
| 17:59:41 | sean-k-mooney | we proably should figure if we can raise the pip/setuptools version in devstack if we really do need a newer versions. that or blacklist pysaml2===4.6.0 | |
| 18:23:58 | openstackgerrit | Merged openstack/nova master: Optimize global marker re-lookup in multi_cell_list https://review.openstack.org/594577 | |
| 18:27:04 | sean-k-mooney | dansmith: by the way, we were talking about deleteing nova-net yesterday. just realisted that we are still blocking dropping rootwrap until nova-net is gone https://review.openstack.org/#/c/554438/ | |
| 18:27:29 | dansmith | sean-k-mooney: aye | |
| 18:27:48 | sean-k-mooney | is this on the ptg techdebt section? ill add it if not | |
| 18:32:27 | sean-k-mooney | there my only addtiontion to the etherpad this time round. suggesting we drop cellsv1 so we can drop nova-net so we can drop oslo-rootwrap. tottally trivial and not controversial at all. at least i hope cern will agree with that else i will be sad | |
| 19:05:33 | openstackgerrit | Eric Fried proposed openstack/nova master: Compute: Handle reshaped provider trees https://review.openstack.org/576236 | |
| 19:28:52 | smcginnis | Anyone aware of any known issues that would be causing this - http://logs.openstack.org/93/596493/1/check/cinder-tempest-dsvm-lvm-lio-barbican/d716acf/logs/screen-n-cpu.txt.gz#_Aug_27_14_31_35_678991 | |
| 19:30:48 | efried | mriedem, dansmith: Didn't y'all see this the other day ^ ? | |
| 19:31:01 | mriedem | no it was an instance.save() when i saw it | |
| 19:32:18 | smcginnis | Odd that it seems to be somewhere that it kills the whole service. | |
| 19:32:25 | mriedem | in this case, compute is asking the cell conductor for the min nova-compute service version in that cell, | |
| 19:32:30 | mriedem | and timing out waiting for a reply | |
| 19:32:40 | mriedem | smcginnis: it's on startup of the service - init the compute rpc api | |
| 19:32:59 | mriedem | i wonder if nova-compute is starting before nova-cond_cell1? | |
| 19:33:09 | mriedem | n-cond-cell1 i mean | |
| 19:33:22 | smcginnis | Was just checking timestamps there. | |
| 19:33:23 | openstackgerrit | Eric Fried proposed openstack/nova master: Do test_reshape with an actual startup https://review.openstack.org/597218 | |
| 19:33:47 | mriedem | n-cond-cell1 appears to be up around at least Aug 27 14:30:40.185973 | |
| 19:34:03 | mriedem | n-cpu fails at Aug 27 14:31:35.678991 | |
| 19:34:16 | smcginnis | Yeah, about a second ahead of the failure on another one I'm looking at too. | |
| 19:34:34 | mriedem | so n-cpu and n-cond-cell1 looks like they are starting at roughly the same time, | |
| 19:34:43 | mriedem | n-cpu starts querying n-cond-cell1 before it's fully up maybe | |
| 19:34:46 | mriedem | then times out | |
| 19:35:27 | smcginnis | Does that need to happen in __init__? Seems like maybe somewhere after where it could retry once or twice might be good. | |
| 19:36:24 | mriedem | doesn't probably have to no | |
| 19:36:29 | mriedem | we could do it in post_start_hook or something | |
| 19:36:32 | mriedem | not sure if it would help | |
| 19:37:15 | mriedem | self.compute_rpcapi is pretty much only used for inter-compute comms anyway | |
| 19:37:20 | mriedem | it's not used on startup | |
| 19:38:32 | mriedem | 2018-08-27 14:30:31.159 | + functions-common:_run_under_systemd:1482 : sudo systemctl start devstack@n-cond-cell1.service | |